the roi of ux design

The ROI of UX DesignUnpacking and measuring the impact of UX design

Werner Janse van RensburgCo-founder, ISOflow

1 The Art of Measuring

2 Data Types

3 Metrics

Four Parts:

4 Case Studies

Part

1The Art of Measuring

(the short version)

Image: http://www.turbosquid.com/3d-models/3d-ruler-model/371813

Image: http://en.wikipedia.org/wiki/History_of_the_metric_systemSource: http://en.wikipedia.org/wiki/Seconds_pendulum

1664: Seconds-Pendulum. Distance for a half-rotation in 1 second

1789: Decimal system introduced. Almost time too (2014-10-03.427 or 42.7% of a day)

Source: http://en.wikipedia.org/wiki/History_of_the_metre

Image: http://www.examiner.com/slideshow/hurricane-arthur-speaks-math-as-well-as-weather

1792–98: One ten-millionth of the distance from the Earth's equator to the North Pole (at sea level).Dunkirk to Barcelona, by Méchain & Delambre.

1799: Mètre des Archives(Platinum Bar)

Source: http://en.wikipedia.org/wiki/History_of_the_metre

Image: http://fr.wikipedia.org/wiki/M%C3%A8tre

Meter standard engraved in marble in 16 locations in Paris

Image: http://en.wikipedia.org/wiki/History_of_the_metric_system

The public was educated on these new standards

Image: http://en.wikipedia.org/wiki/History_of_the_metre

1875: BIPM and the International Prototype Meter (Platinum-Iradium)

The unit of length is the metre, defined by the distance, at 0°, between the axes of the two central lines marked on the bar of platinum–iridium kept at the Bureau International des Poids et Mesures and declared Prototype of the metre by the 1st Conférence Générale des Poids et Mesures, this bar being subject to standard atmospheric pressure and supported on two cylinders of at least one centi- metre diameter, symmetrically placed in the same horizontal plane at a distance of 571 mm from each other.

1927: 7th BIPM General Conference:

Define Source

,,,,

1960: Krypton standard and the interferometer.Later used laser, in 1975.

Source: http://en.wikipedia.org/wiki/History_of_the_metre

The length of the path travelled by light in vacuum during a time interval of 1/299,792,458 of a second

,,

,,

1983: Speed of light

Why am I telling you all this?

We can all use a ruler

We can all measure UX

Source: http://www.usernomics.com/iaa_aed_2003.pdfImage: http://defibuk.co.uk

Three defining characteristics:

• A user is involved• That user is interacting with a

product, system, or really anything with an interface

• The users’ experience is of interest, and observable or measurable

Source: http://www.usernomics.com/iaa_aed_2003.pdfImage: http://defibuk.co.uk

Automatic External Defribulators

• Used by untrained persons in public settings.• Four devices tested by 64 users (16 per device).• Two devices were successful 100% of the time. • 9 / 16 (56%) and 4 / 16 users (25%)

FAILED on the remaining two devices

Why is this important

for us?

Part

2Data Types

The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.

ISO 9241-11:1998 (Usability)

,,,,

ISO 9241-11:1998 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability

UX Metrics reveals something about:

• Effectiveness– “Being able to complete a task”

• Efficiency– “The amount of effort required to complete the

task”• Satisfaction– “The degree to which the user was happy with his

or her experience while performing the task”

Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Approach: Formative Summative

Image: http://portandfin.com/spaghetti-with-mango-roasted-peppers-basil/

Qualitative Data• Deals with descriptions• Data can be observed but not

measured• Colors, appearance, beauty,

smells, tastes etc.• Qualitative > Quality

Quantitative Data• Deals with numbers• Data which can be measured• Length, height, area, volume,

weight, speed, time, temperature, humidity, sound levels, cost, members, ages etc.

• Quntitative > Quanitity

Image: http://www.artchive.com/meninas.htm Source: http://regentsprep.org/regents/math/algebra/AD1/qualquant.htm

• Gold frame• Smells Old / Musty• Nice Texture• Brush Strokes• Masterful

• Art: 100 x 140cm• Frame: 120x160cm• Weight: 8.5kg• Surface: 14,000cm2

• Cost: $3m

• Comprehensive look at UX Metrics

• Practical Approach• Help you make the

right decisions• Examples• Relevant to many

products and technologies

Helps to answer critical questions:

• “Will the users recommend the product?• Is this new product more efficient to use than the

current product?”• How does the user experience of this product

compare to the competition? • What are the most significant usability problems

with this product? • Are improvements being made from one design

iteration to the next?”• … as well as avoid wrong assumptions.

Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

It’s not the things we don’t know that gets us in trouble, it’s the things we do know that ain’t so.

,,,,

To Assumeis to make an

ass out of u and me.

Types of Data

Nominal Ordinal Interval Ratio

Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Nominal Data

– Just a grouping– Fruit / Gender / Eye Color– Type of Car– Statistics:• 45% of users are female• 25% preferred apples• 200 users had blue eyes

Image: http://citysportsblog.com/city-sports/fruit-how-much-is-too-much/Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Ordinal Data

– Ordered data– Intervals not meaningful– Order from best to worse– Rate this site

– Statistics:• 40% rates this site “Good” • 68% Prefer Option A• Rated 28th on IMDB Top 250

Poor Fair Good Excellent

28

56

Image: http://www.imdb.com/chart/topSource: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Interval Data

– Differences meaningful– There’s no natural 0– Temperature, Dates– System Usability Scale (SUS)– Statistics:• Increase / decrease in usability (%)• Averages / Standard deviation

Image: http://envirodailyadvisor.blr.com/2013/05/tips-for-safe-mercury-cleanup/ & https://www.iconfinder.com Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Ratio Data

– Similar to interval data– With an absolute 0– E.g. Time, Age, Weight, Height– Zero has inherent meaning

(Absence of Age / Weight)– Or no time remaining– Statistics:• Twice as fast or half as slow

Image: http://www.softwareag.com/ & https://www.iconfinder.com Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Descriptive Statistics

• Describes the data, without saying anything about the larger population

• Most common measures used:– Central Tendency – Variability– Confidence Intervals

Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Measures of Central Tendency

• Mean / Average• Sum / # of items• 243/10 = 24.3

• Median • Middle number when ordered

from small to large. Half of values are above,half below.

• 17 (between 16 & 18)

Participant Task Time (Seconds)

P1 11

P2 18

P3 12

P4 29

P5 10

P6 53

P7 16

P8 14

P9 22

P10 58=AVARAGE function=MEAN function

Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Participant Task Time (Seconds)

P1 11

P2 18

P3 12

P4 29

P5 10

P6 53

P7 16

P8 14

P9 22

P10 58

Measures of Variability

• Range• Range = Max - Min• 58-10 = 48

• Variance• 303.8

• Standard Deviation• Square root of the variance• 17.4 Seconds

=MAX and =MIN functions=VAR and =STDEV functions

Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Confidence Intervals

• Confidence Level: 95%• Want to be 95% Certain• Willing to be wrong 5% of

time (Alpha)• Most commonly used: • Confidence 99%, 95%, 90%)

(Alpha 1%, 5%, 10%)• E.g. 1% for AED tests

=CONFIDENCE(alpha, stdev, sample size)

Design A Design B0

20

40

60

80

100

120

Mean Checkout Time (Seconds)

(Error Bars Rep-resent 90% con-

fidence intervals)

Chec

kout

Tim

e (S

econ

ds)

Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Part

3Metrics

(and how to render results)

Categories:

• Performance Metrics• Issue Based Metrics• Self Reported Metrics• Behavioral and Physiological Metrics• Live Website Data

Performance Metrics

• Task Success• Time on Task• Errors• Efficiency• Learnability

UX

Column / Bar Graphs

Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Line Graphs

Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Stacked Bar / Column Graphs

Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Issue-Based Metrics

• Severity• Frequency• By Category• By Task

UX

Pie or Donut Charts

Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Self-Reported Metrics

• Rating Scales• Ease of Use• System Usability Scale• Satisfaction • Open Ended Questions

UX

Scatterplots

Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Source: http://ebusinesshow.blogspot.com/2011/02/third-weekly-task-usability-evaluation.html

62.5%

Behavioral and Physiological Metrics

• Eye Tracking• Emotion & Stress

UX

Image: http://usableworld.com.au/2009/03/16/you-look-where-they-look/

Image: http://mashable.com/2014/05/19/tobii-glasses/

Image: https://www.flickr.com/photos/56422507@N07/5827330051/

Live Website Data

• Basic Web Analytics • Click-Through Rates • Drop-Off Rates • A/ B Tests

UX

Image: http://www.thesearchagents.com/2012/04/how-to-create-a-mobile-marketing-dashboard-in-google-analytics/

Part

4Case Studies

(Gains – Cost)

CostROI =

Every $1 invested in usability returns between $10 and $100.

,,,,

IBM, Cost-Justifying Usability

And if we don’t…

There’s training, support and maintenance costs. Costs associated with fixing problems increase:

• $1 to fix in early analysis• $10 to fix in design• $100 to fix in a prototype• $1000 to fix after deployment

Mantei and Teorey (1988)

We don’t always measure $

Intranet E-commerce AppWebsites (Content / Marketing)

Productivity

Time saved x

hourly cost of employees

Revenue

Drop-off rates

Conversion Rates

Customer Satisfaction

Increased Feature use

Visitors

Pageviews

Customer Loyalty

Sales Impact

10%Of project budget

The cost of usability:

Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition

Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition

72

Usability Metrics

Benefits from usability

160%2.6x

72

Usability Metrics

Benefits from usability

109%2.1x

With 8 outliers excluded

Only one metric, before & after:

• Conversions• Request for Quote• Sales• Reservations• Help desk calls• Newsletter signups• Trial Signups• Time on Site

• Reduced Exits• Traffic • Return Visitors• Feature Use• Comments• Customer Satisfaction• Task Time• Success Rate

Improvement in Usability Metrics

Metric Average Improvement Across Web Projects

Sales / conversion rate 87%

Traffic / visitor count 91%

User performance / productivity 112%

Use of specific (desired) features 174%

Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition

Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition

Ways to double your business

B = V x C x LB = amount of business done by the siteV = unique visitors coming to the siteC = conversion rate L = loyaltyC = conversion rate

2000-2010 Conversion Decade

L = loyalty2010-2020

Loyalty Decade

Skilled-user Performance

Error-Avoidance

More users, bigger cost impact

• Guest Checkout• One Click Purchase• User Ratings

Investment would increase further

• Competition• Return on Investment• Changing Audience• Internationalization

Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition

Eurostar

• Usability & IA• New booking features• New Brand Identity• Market-leading experience• Six-month timeframe

Metric: Online sales (12 mnths)

Before: £110 million/year

After: £136 million/year

Ratio: 124%Improvement: 24%

Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Eurostar

• Issues resolved:– Error Messages– Confusing Language– Confirmation Pages– User Accounts– HTML Issues

Metric: Online sales (12 mnths)

Before: £110 million/year

After: £136 million/year

Ratio: 124%Improvement: 24%

Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition

100+ Issues

70+ Issues

High-fidelity

Adobe Kuler

• Color Theme Sharing Site• Comment Engagement low• Unconventional Display

Metric: # of Comments

Before: 6 / day (Average)

After: 37 / day (Average)

Ratio: 617%Improvement: 517%

Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition

Familiarity does not necessarily breed contempt in Web design. Unless a feature offers entirely new functionality or much improved intuitive use, it is often better to stick with a conventional delivery.

,,,,

Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition

In closing…

Remote Usability Testing:• http://www.optimalworkshop.com/

chalkmark.htm• http://ethn.io/• http://www.feedbackarmy.com/• http://fivesecondtest.com/• http://www.keynote.com/• http://www.loop11.com/• http://www.trymyui.com/• http://usabilitytools.com/• http://www.usabilla.com/• http://www.userlytics.com/• http://www.usertesting.com/

Offline Usability Testing:• http://silverbackapp.com/• http://www.techsmith.com/

camtasia.html• http://www.techsmith.com/

morae.html

A/B Testing:• http://www.adobe.com/africa/solutio

ns/testing-targeting.html

• http://useartisan.com/• http://www.globalmaxer.com/• http://optimizely.com/• https://vwo.com/

Source: http://uxpin.com/upload/ux-design-for-startups-marcin-treder.pdf

Tools, tools, tools…

ClickTracking:• http://www.labsmedia.com/clickheat

/index.html

• http://www.clicktale.com/• http://www.crazyegg.com/• http://usabilitytools.com/features-be

nefits/click-tracking/#tool-description

• http://www.userzoom.com/

Eye Tracking:• http://eyetrackshop.com/

Session Recording:• http://www.userzoom.com/• http://www.ghostrec.com/• http://mouseflow.com/• http://www.openhallway.com/• http://www.tealeaf.com/• http://usabilitytools.com/features-be

nefits/visitor-recording/#tool-description

• http://userreplay.co.uk/

Source: http://uxpin.com/upload/ux-design-for-startups-marcin-treder.pdf

Tools, tools, tools… (more)

Keys to successto help you with your UX measurements

• Make Data Come Alive • Don’t Wait to be Asked to Measure • Measurement is Less Expensive Than You Think • Plan Early • Benchmark Your Products • Explore Your Data • Speak the Language of Business • Show Your Confidence • Don’t Misuse Metrics• Simplify Your Presentation

10x

Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition

Image: http://fr.wikipedia.org/wiki/M%C3%A8tre

Thanks!

Werner Janse van Rensburg@wernerjvr@isoflowwww.isoflow.co.za