measuring the wrong thing: data-driven design pitfalls

Measuring the Wrong Thing:Data-Driven Design Pitfalls

@nstop Jen Matson

Hi, I’m Jen Matson.

• Senior UX Designer at Amazon• Designing & building web sites since 1994• Unabashed data junkie

!"#"$%&'

('&"$%&'

!)*)#"%+,-'./#'&0)*1

2&3'&0)*1

From Ideas and/or data by Cennydd Bowles

Case Study #1:The Meaning of A Click (or Tap)

Case Study #1:The Meaning of A Click (or Tap)

Company:Movie listings site

Project:Create a mobile-optimized view of the movie detail page

Case Study #1:The Meaning of A Click (or Tap)

Movie Showtimes

Average Review

Case Study #1:The Meaning of A Click (or Tap)

Movie Showtimes

Average Review

Case Study #1:The Meaning of A Click (or Tap)

Movie Showtimes

Average Review

Result:Mistaking one thing for another

Causes:• Too focused on quantitative data• Clickable UI elements too

closely grouped

Case Study #1:The Meaning of A Click (or Tap)

Movie Showtimes

Average Review

Potential impact:• Frustrated users due to

“broken” UI• Drawing the wrong conclusions

about what users want/like• Building more features based

on those conclusions

Case Study #1:The Meaning of A Click (or Tap)

Movie Showtimes

Average Review

How to fix:• Gather qualitative data

(customer feedback) along with quantitative

• Make time for usability testing, and subsequent design/dev cycle prior to launch

Case Study #2:Throwing Stuff Against the Wall

Case Study #2:Throwing Stuff Against the Wall

Company:Mobile service provider site

Project:Redesign the help portal to offer personalized content

Case Study #2:Throwing Stuff Against the Wall

Help

Case Study #2:Throwing Stuff Against the Wall

Help

Case Study #2:Throwing Stuff Against the Wall

Help

Case Study #2:Throwing Stuff Against the Wall

Help

Result:False positive

Causes:• Choosing a metric

(clicks) with only a loose connection to user need

• Poor communication between teams

Case Study #2:Throwing Stuff Against the Wall

Help

Impact:• Irrelevant content

leads to user confusion, lack of trust

• Failure to improve help relevance due to bad data feedback loop

Case Study #2:Throwing Stuff Against the Wall

Help

How to fix:• Audit relevance of

help, match to real user attributes

• Use real user events to power suggestions

• Unify project teams

Case Study #3:Unclear Cause and Effect

Case Study #3:Unclear Cause and Effect

Company:TV manufacturer site

Project:Redesign the search engine for the support section to make content easier to find

Case Study #3:Unclear Cause and Effect

User tasks:1.Find article (search)

2.Read article

3.Use solution or tool found in article to solve problem

Case Study #3:Unclear Cause and Effect

User tasks:1.Find article (search)

Content: Findable

2.Read articleContent: Consumable

3.Use solution or tool found in article to solve problemContent: Actionable

Case Study #3:Unclear Cause and Effect

Search Results

Case Study #3:Unclear Cause and Effect

Search Results

Contact

Case Study #3:Unclear Cause and Effect

Search Results

Help Article Tool

Contact

Case Study #3:Unclear Cause and Effect

Search Results

Help Article Tool

Contact

Case Study #3:Unclear Cause and Effect

Search Results

Result:Unclear impact

Causes:• Choosing to measure only what we

were already set up to measure• Lack of data to ensure business

goals are aligned with project work

Case Study #3:Unclear Cause and Effect

Search Results

Impact:• Data gathered from product

launch not useful in helping to prioritize future features

• Further defer updates to content and tools due to lack of data

Case Study #3:Unclear Cause and Effect

Search Results

How to fix:• Work with product manager on

goals, project definition before finalized

• Use customer journey and task mapping to highlight data collection needs

What else?

Understand your company culture

Learn more about what you can measure,

and how

Learn more about what you can measure,

and how

hovers

clicks

scroll depth

site path

mouse pathdwell time

data inputs

Use what you learn to improve your designs and

increase your influence

Add new questions to your

arsenal

Add new questions to your

arsenal

What data do we have to support this?

How will we get data to validate this?

Thank you.

@nstop Jen Matson

https://www.flickr.com/photos/72764087@N00/9990024683/https://www.flickr.com/photos/coolmel/5469163/https://www.flickr.com/photos/37182073@N06/5142618640/

Photo credits (in order of appearance):https://www.flickr.com/photos/notemily/4765937286https://www.flickr.com/photos/gwdexter/1401789875