The traditional role of user researcher was killed off by new technologies like usability test automation, remote data collection and machine learning. These days, we have minute-by-minute breakdowns of aggregate user behaviour that we can splice and dice in anyway we want. Want to see what tech-averse millennials in the South West of England are up to? Click this button. Ethnographic research—and researchers—became marginalised by new forms of automated data and insight generation, compiled via remote sensing and delivered to designers through technologies like virtual reality. To go out on a field visit these days I sit at my desk and slip on my 3-D headset.

Fat data

Skinny data

Res

earc

h m

edia

ted

by a

UX

rese

arch

erAutom

ated research

2003 Motorola i95cl

Fat data

Skinny data

Res

earc

h m

edia

ted

by a

UX

rese

arch

erAutom

ated researchSurveys

Diary studies

Card sorting

Telephone interviews

Web analytics

Formative usability testing

Ethnographic field

Face-to-face interviews Pop-up research

Summative usability 2003

Fat data

Skinny data

Res

earc

h m

edia

ted

by a

UX

rese

arch

erAutom

ated researchSurveys

Diary studies

Card sorting

Telephone interviews

Web analytics

Formative usability

Ethnographic field

Face-to-face interviews Pop-up research

Summative usability 2003

2003 2019

Fat data

Skinny data

Res

earc

h m

edia

ted

by a

UX

rese

arch

erAutom

ated researchSurveys

Diary studies

Card sorting

Telephone interviews

Web analytics

Formative usability testing

Ethnographic field

Face-to-face interviews Pop-up research

Summative usability 2003

2019Fat data

Skinny data

Res

earc

h m

edia

ted

by a

UX

rese

arch

erAutom

ated researchSurveys

Diary studies

Card sorting

Telephone interviews

Web analytics

Formative usability testing

Ethnographic field

Face-to-face interviews Pop-up research

Summative usability

Eye tracking

Moderated remote usability studies

Clickstream analysis

A/B testing 5-second tests

2019Fat data

Skinny data

Res

earc

h m

edia

ted

by a

UX

rese

arch

erAutom

ated research

Surveys

Diary studies

Card sorting

Telephone interviews

Web analytics

Formative usability testing

Ethnographic field

Face-to-face interviews Pop-up research

Summative usability

Eye tracking

Moderated remote usability studies

Clickstream analysis

A/B testing 5-second tests

Why the future of UX Research is automated

Quick to set up

Cheap to commission

Generates skinny data

Don’t have to speak with users

Easy to recruit

Automated research blocks

your view

Behaviour

Design

Behaviour

Design

Report date: 20th Sept 2006. Test date: 11th-15th Sept 2006.

+ Several quick search options are provided Sample size: 17

+ Landing pages are clear and usable 59% Male, 41% Female

+ Homepage search provides helpful quick links 47% 18-35, 47% 36+

+ Breadcrumbs show user journey Mobile providers

- List of phone features is limited

- Accessory purchase is very problematic T-Mobile

- Users don't notice the price change on 12-month contracts Vodafone

- Users do not see the “order now” call to action O2

Desirability

Detailed TCR Detailed SOU

Speed of use (SOU)

Detailed Desirability

Private and Confidential • For more details about the data in this report, contact David Travis • dave.travis@orange.co.uk • Design © Userfocus 2006

Web site is equal to best competitor

TCR: Competitors SOU: Competitors Desirability: Competitors

StatusStrengths & Weaknesses Demographics

Competitor benchmarks

29%

Web Usability Dashboard This report compares the usability of vodafone.co.uk with three competitors

Margin of error: ±8.9%

Task completion (TCR)

64.1% 54.4% 61.4%Overall Benchmark

60.9%

PAYM

Orange

Buy a 3G phone, 12-month contract and Bluetooth headset

Buy the cheapest PAYG phone with internet and MP3

24%

24%

24%

PAYG

59%

69%

0% 25% 50% 75% 100%

PAY

MPA

YG

52%

40%

59%61%

0%

25%

50%

75%

100%

Voda Or O2 T-M

64%54%

65%

43%

0%

25%

50%

75%

100%

Voda Or O2 T-M

54%

40%

54%

39%

0%

25%

50%

75%

100%

Voda Or O2 T-M

61%

73%

40%

26%

0%

25%

50%

75%

100%

Voda Or O2 T-M

81%

39%

0% 25% 50% 75% 100%

PAY

MPA

YG

0%

25%

50%

75%

100%

Q1 Q10

Report date: 20th Sept 2006. Test date: 11th-15th Sept 2006.

+ Several quick search options are provided Sample size: 17

+ Landing pages are clear and usable 59% Male, 41% Female

+ Homepage search provides helpful quick links 47% 18-35, 47% 36+

+ Breadcrumbs show user journey Mobile providers

- List of phone features is limited

- Accessory purchase is very problematic T-Mobile

- Users don't notice the price change on 12-month contracts Vodafone

- Users do not see the “order now” call to action O2

Desirability

Detailed TCR Detailed SOU

Speed of use (SOU)

Detailed Desirability

Private and Confidential • For more details about the data in this report, contact David Travis • dave.travis@orange.co.uk • Design © Userfocus 2006

Web site is equal to best competitor

TCR: Competitors SOU: Competitors Desirability: Competitors

StatusStrengths & Weaknesses Demographics

Competitor benchmarks

29%

Web Usability Dashboard This report compares the usability of vodafone.co.uk with three competitors

Margin of error: ±8.9%

Task completion (TCR)

64.1% 54.4% 61.4%Overall Benchmark

60.9%

PAYM

Orange

Buy a 3G phone, 12-month contract and Bluetooth headset

Buy the cheapest PAYG phone with internet and MP3

24%

24%

24%

PAYG

59%

69%

0% 25% 50% 75% 100%

PAY

MPA

YG

52%

40%

59%61%

0%

25%

50%

75%

100%

Voda Or O2 T-M

64%54%

65%

43%

0%

25%

50%

75%

100%

Voda Or O2 T-M

54%

40%

54%

39%

0%

25%

50%

75%

100%

Voda Or O2 T-M

61%

73%

40%

26%

0%

25%

50%

75%

100%

Voda Or O2 T-M

81%

39%

0% 25% 50% 75% 100%

PAY

MPA

YG

0%

25%

50%

75%

100%

Q1 Q10

— Paul Brodeur

“Statistics are human beings with the tears wiped away.”

Formative usability testing

Fat data

Skinny data

Res

earc

h m

edia

ted

by a

UX

rese

arch

erAutom

ated research

Surveys

Diary studies

Card sorting

Telephone interviews

Web analytics

Formative usability testing

Ethnographic field

Face-to-face interviews Pop-up research

Summative usability

Eye tracking

Moderated remote usability studies

Clickstream analysis

A/B testing 5-second tests

These research methods drive

empathy

Formative usability testing

Fat data

Skinny data

Res

earc

h m

edia

ted

by a

UX

rese

arch

erAutom

ated research

Surveys

Diary studies

Card sorting

Telephone interviews

Web analytics

Formative usability testing

Ethnographic field

Face-to-face interviews Pop-up research

Summative usability

Eye tracking

Moderated remote usability studies

Clickstream analysis

A/B testing 5-second testsAutomated

research methods do not drive

empathy

Good quality UX research is not about: - Collecting numbers- Statistical significance- Sample sizeGood quality UX research is research that provides actionable and testable insights into users’ needs.

The local maxima problem in UX research

Metrics-Driven Design SXSWThe grass is always greener on other mountainsides.

Imagine that your design is a mountain...

Saturday, March 12, 2011

Image credit: https://www.slideshare.net/andrew_null/metrics-driven-design-by-joshua-porter

Design 1

Design 1 - Optimised

Design 2 - Unexplored

Small difference

Big difference

A successful design process doesn’t start with Build.

It starts with Learn.

The pleasures of cystoscopy

Me

A sensitive part of my anatomy

— Response scale from 0 - 10

“How likely is it that you would recommend the Urology Department of this hospital to a friend or colleague?”

You are not the user.

In summary Unless you take action, UX research will become increasingly automated.Automated research blocks your team’s view of the user.Encourage your development team to triangulate automated research with methods that create empathy.