Measuring the Wrong Thing:Data-Driven Design Pitfalls
@nstop Jen Matson
Hi, I’m Jen Matson.
• Senior UX Designer at Amazon• Designing & building web sites since 1994• Unabashed data junkie
!"#"$%&'
('&"$%&'
!)*)#"%+,-'./#'&0)*1
2&3'&0)*1
From Ideas and/or data by Cennydd Bowles
Case Study #1:The Meaning of A Click (or Tap)
Case Study #1:The Meaning of A Click (or Tap)
Company:Movie listings site
Project:Create a mobile-optimized view of the movie detail page
Case Study #1:The Meaning of A Click (or Tap)
Movie Showtimes
Average Review
Case Study #1:The Meaning of A Click (or Tap)
Movie Showtimes
Average Review
Case Study #1:The Meaning of A Click (or Tap)
Movie Showtimes
Average Review
Result:Mistaking one thing for another
Causes:• Too focused on quantitative data• Clickable UI elements too
closely grouped
Case Study #1:The Meaning of A Click (or Tap)
Movie Showtimes
Average Review
Potential impact:• Frustrated users due to
“broken” UI• Drawing the wrong conclusions
about what users want/like• Building more features based
on those conclusions
Case Study #1:The Meaning of A Click (or Tap)
Movie Showtimes
Average Review
How to fix:• Gather qualitative data
(customer feedback) along with quantitative
• Make time for usability testing, and subsequent design/dev cycle prior to launch
Case Study #2:Throwing Stuff Against the Wall
Case Study #2:Throwing Stuff Against the Wall
Company:Mobile service provider site
Project:Redesign the help portal to offer personalized content
Case Study #2:Throwing Stuff Against the Wall
Help
Case Study #2:Throwing Stuff Against the Wall
Help
Case Study #2:Throwing Stuff Against the Wall
Help
Case Study #2:Throwing Stuff Against the Wall
Help
Result:False positive
Causes:• Choosing a metric
(clicks) with only a loose connection to user need
• Poor communication between teams
Case Study #2:Throwing Stuff Against the Wall
Help
Impact:• Irrelevant content
leads to user confusion, lack of trust
• Failure to improve help relevance due to bad data feedback loop
Case Study #2:Throwing Stuff Against the Wall
Help
How to fix:• Audit relevance of
help, match to real user attributes
• Use real user events to power suggestions
• Unify project teams
Case Study #3:Unclear Cause and Effect
Case Study #3:Unclear Cause and Effect
Company:TV manufacturer site
Project:Redesign the search engine for the support section to make content easier to find
Case Study #3:Unclear Cause and Effect
User tasks:1.Find article (search)
2.Read article
3.Use solution or tool found in article to solve problem
Case Study #3:Unclear Cause and Effect
User tasks:1.Find article (search)
Content: Findable
2.Read articleContent: Consumable
3.Use solution or tool found in article to solve problemContent: Actionable
Case Study #3:Unclear Cause and Effect
Search Results
Case Study #3:Unclear Cause and Effect
Search Results
Contact
Case Study #3:Unclear Cause and Effect
Search Results
Help Article Tool
Contact
Case Study #3:Unclear Cause and Effect
Search Results
Help Article Tool
Contact
Case Study #3:Unclear Cause and Effect
Search Results
Result:Unclear impact
Causes:• Choosing to measure only what we
were already set up to measure• Lack of data to ensure business
goals are aligned with project work
Case Study #3:Unclear Cause and Effect
Search Results
Impact:• Data gathered from product
launch not useful in helping to prioritize future features
• Further defer updates to content and tools due to lack of data
Case Study #3:Unclear Cause and Effect
Search Results
How to fix:• Work with product manager on
goals, project definition before finalized
• Use customer journey and task mapping to highlight data collection needs
What else?
Understand your company culture
Learn more about what you can measure,
and how
Learn more about what you can measure,
and how
hovers
clicks
scroll depth
site path
mouse pathdwell time
data inputs
Use what you learn to improve your designs and
increase your influence
Add new questions to your
arsenal
Add new questions to your
arsenal
What data do we have to support this?
How will we get data to validate this?
Thank you.
@nstop Jen Matson
https://www.flickr.com/photos/72764087@N00/9990024683/https://www.flickr.com/photos/coolmel/5469163/https://www.flickr.com/photos/37182073@N06/5142618640/
Photo credits (in order of appearance):https://www.flickr.com/photos/notemily/4765937286https://www.flickr.com/photos/gwdexter/1401789875
