the roi of ux design
DESCRIPTION
Unpacking and measuring the impact of UX design. A talk presented at the inaugural UX South Africa Conference on 3 September 2014.TRANSCRIPT
The ROI of UX DesignUnpacking and measuring the impact of UX design
Werner Janse van RensburgCo-founder, ISOflow
1 The Art of Measuring
2 Data Types
3 Metrics
Four Parts:
4 Case Studies
Part
1The Art of Measuring
(the short version)
Image: http://www.turbosquid.com/3d-models/3d-ruler-model/371813
Image: http://en.wikipedia.org/wiki/History_of_the_metric_systemSource: http://en.wikipedia.org/wiki/Seconds_pendulum
1664: Seconds-Pendulum. Distance for a half-rotation in 1 second
1789: Decimal system introduced. Almost time too (2014-10-03.427 or 42.7% of a day)
Source: http://en.wikipedia.org/wiki/History_of_the_metre
Image: http://www.examiner.com/slideshow/hurricane-arthur-speaks-math-as-well-as-weather
1792–98: One ten-millionth of the distance from the Earth's equator to the North Pole (at sea level).Dunkirk to Barcelona, by Méchain & Delambre.
1799: Mètre des Archives(Platinum Bar)
Source: http://en.wikipedia.org/wiki/History_of_the_metre
Image: http://fr.wikipedia.org/wiki/M%C3%A8tre
Meter standard engraved in marble in 16 locations in Paris
Image: http://en.wikipedia.org/wiki/History_of_the_metric_system
The public was educated on these new standards
Image: http://en.wikipedia.org/wiki/History_of_the_metre
1875: BIPM and the International Prototype Meter (Platinum-Iradium)
The unit of length is the metre, defined by the distance, at 0°, between the axes of the two central lines marked on the bar of platinum–iridium kept at the Bureau International des Poids et Mesures and declared Prototype of the metre by the 1st Conférence Générale des Poids et Mesures, this bar being subject to standard atmospheric pressure and supported on two cylinders of at least one centi- metre diameter, symmetrically placed in the same horizontal plane at a distance of 571 mm from each other.
1927: 7th BIPM General Conference:
Define Source
,,,,
1960: Krypton standard and the interferometer.Later used laser, in 1975.
Source: http://en.wikipedia.org/wiki/History_of_the_metre
The length of the path travelled by light in vacuum during a time interval of 1/299,792,458 of a second
,,
,,
1983: Speed of light
Why am I telling you all this?
We can all use a ruler
We can all measure UX
Source: http://www.usernomics.com/iaa_aed_2003.pdfImage: http://defibuk.co.uk
Three defining characteristics:
• A user is involved• That user is interacting with a
product, system, or really anything with an interface
• The users’ experience is of interest, and observable or measurable
Source: http://www.usernomics.com/iaa_aed_2003.pdfImage: http://defibuk.co.uk
Automatic External Defribulators
• Used by untrained persons in public settings.• Four devices tested by 64 users (16 per device).• Two devices were successful 100% of the time. • 9 / 16 (56%) and 4 / 16 users (25%)
FAILED on the remaining two devices
Why is this important
for us?
Part
2Data Types
The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.
ISO 9241-11:1998 (Usability)
,,,,
ISO 9241-11:1998 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability
UX Metrics reveals something about:
• Effectiveness– “Being able to complete a task”
• Efficiency– “The amount of effort required to complete the
task”• Satisfaction– “The degree to which the user was happy with his
or her experience while performing the task”
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Approach: Formative Summative
Image: http://portandfin.com/spaghetti-with-mango-roasted-peppers-basil/
Qualitative Data• Deals with descriptions• Data can be observed but not
measured• Colors, appearance, beauty,
smells, tastes etc.• Qualitative > Quality
Quantitative Data• Deals with numbers• Data which can be measured• Length, height, area, volume,
weight, speed, time, temperature, humidity, sound levels, cost, members, ages etc.
• Quntitative > Quanitity
Image: http://www.artchive.com/meninas.htm Source: http://regentsprep.org/regents/math/algebra/AD1/qualquant.htm
• Gold frame• Smells Old / Musty• Nice Texture• Brush Strokes• Masterful
• Art: 100 x 140cm• Frame: 120x160cm• Weight: 8.5kg• Surface: 14,000cm2
• Cost: $3m
• Comprehensive look at UX Metrics
• Practical Approach• Help you make the
right decisions• Examples• Relevant to many
products and technologies
Helps to answer critical questions:
• “Will the users recommend the product?• Is this new product more efficient to use than the
current product?”• How does the user experience of this product
compare to the competition? • What are the most significant usability problems
with this product? • Are improvements being made from one design
iteration to the next?”• … as well as avoid wrong assumptions.
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
It’s not the things we don’t know that gets us in trouble, it’s the things we do know that ain’t so.
,,,,
To Assumeis to make an
ass out of u and me.
Types of Data
Nominal Ordinal Interval Ratio
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Nominal Data
– Just a grouping– Fruit / Gender / Eye Color– Type of Car– Statistics:• 45% of users are female• 25% preferred apples• 200 users had blue eyes
Image: http://citysportsblog.com/city-sports/fruit-how-much-is-too-much/Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Ordinal Data
– Ordered data– Intervals not meaningful– Order from best to worse– Rate this site
– Statistics:• 40% rates this site “Good” • 68% Prefer Option A• Rated 28th on IMDB Top 250
Poor Fair Good Excellent
28
56
Image: http://www.imdb.com/chart/topSource: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Interval Data
– Differences meaningful– There’s no natural 0– Temperature, Dates– System Usability Scale (SUS)– Statistics:• Increase / decrease in usability (%)• Averages / Standard deviation
Image: http://envirodailyadvisor.blr.com/2013/05/tips-for-safe-mercury-cleanup/ & https://www.iconfinder.com Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Ratio Data
– Similar to interval data– With an absolute 0– E.g. Time, Age, Weight, Height– Zero has inherent meaning
(Absence of Age / Weight)– Or no time remaining– Statistics:• Twice as fast or half as slow
Image: http://www.softwareag.com/ & https://www.iconfinder.com Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Descriptive Statistics
• Describes the data, without saying anything about the larger population
• Most common measures used:– Central Tendency – Variability– Confidence Intervals
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Measures of Central Tendency
• Mean / Average• Sum / # of items• 243/10 = 24.3
• Median • Middle number when ordered
from small to large. Half of values are above,half below.
• 17 (between 16 & 18)
Participant Task Time (Seconds)
P1 11
P2 18
P3 12
P4 29
P5 10
P6 53
P7 16
P8 14
P9 22
P10 58=AVARAGE function=MEAN function
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Participant Task Time (Seconds)
P1 11
P2 18
P3 12
P4 29
P5 10
P6 53
P7 16
P8 14
P9 22
P10 58
Measures of Variability
• Range• Range = Max - Min• 58-10 = 48
• Variance• 303.8
• Standard Deviation• Square root of the variance• 17.4 Seconds
=MAX and =MIN functions=VAR and =STDEV functions
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Confidence Intervals
• Confidence Level: 95%• Want to be 95% Certain• Willing to be wrong 5% of
time (Alpha)• Most commonly used: • Confidence 99%, 95%, 90%)
(Alpha 1%, 5%, 10%)• E.g. 1% for AED tests
=CONFIDENCE(alpha, stdev, sample size)
Design A Design B0
20
40
60
80
100
120
Mean Checkout Time (Seconds)
(Error Bars Rep-resent 90% con-
fidence intervals)
Chec
kout
Tim
e (S
econ
ds)
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Part
3Metrics
(and how to render results)
Categories:
• Performance Metrics• Issue Based Metrics• Self Reported Metrics• Behavioral and Physiological Metrics• Live Website Data
Performance Metrics
• Task Success• Time on Task• Errors• Efficiency• Learnability
UX
Column / Bar Graphs
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Line Graphs
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Stacked Bar / Column Graphs
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Issue-Based Metrics
• Severity• Frequency• By Category• By Task
UX
Pie or Donut Charts
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Self-Reported Metrics
• Rating Scales• Ease of Use• System Usability Scale• Satisfaction • Open Ended Questions
UX
Scatterplots
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Source: http://ebusinesshow.blogspot.com/2011/02/third-weekly-task-usability-evaluation.html
62.5%
Behavioral and Physiological Metrics
• Eye Tracking• Emotion & Stress
UX
Image: http://usableworld.com.au/2009/03/16/you-look-where-they-look/
Image: http://mashable.com/2014/05/19/tobii-glasses/
Image: https://www.flickr.com/photos/56422507@N07/5827330051/
Live Website Data
• Basic Web Analytics • Click-Through Rates • Drop-Off Rates • A/ B Tests
UX
Image: http://www.thesearchagents.com/2012/04/how-to-create-a-mobile-marketing-dashboard-in-google-analytics/
Part
4Case Studies
(Gains – Cost)
CostROI =
Every $1 invested in usability returns between $10 and $100.
,,,,
IBM, Cost-Justifying Usability
And if we don’t…
There’s training, support and maintenance costs. Costs associated with fixing problems increase:
• $1 to fix in early analysis• $10 to fix in design• $100 to fix in a prototype• $1000 to fix after deployment
Mantei and Teorey (1988)
We don’t always measure $
Intranet E-commerce AppWebsites (Content / Marketing)
Productivity
Time saved x
hourly cost of employees
Revenue
Drop-off rates
Conversion Rates
Customer Satisfaction
Increased Feature use
Visitors
Pageviews
Customer Loyalty
Sales Impact
10%Of project budget
The cost of usability:
Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition
Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition
72
Usability Metrics
Benefits from usability
160%2.6x
72
Usability Metrics
Benefits from usability
109%2.1x
With 8 outliers excluded
Only one metric, before & after:
• Conversions• Request for Quote• Sales• Reservations• Help desk calls• Newsletter signups• Trial Signups• Time on Site
• Reduced Exits• Traffic • Return Visitors• Feature Use• Comments• Customer Satisfaction• Task Time• Success Rate
Improvement in Usability Metrics
Metric Average Improvement Across Web Projects
Sales / conversion rate 87%
Traffic / visitor count 91%
User performance / productivity 112%
Use of specific (desired) features 174%
Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition
Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition
Ways to double your business
B = V x C x LB = amount of business done by the siteV = unique visitors coming to the siteC = conversion rate L = loyaltyC = conversion rate
2000-2010 Conversion Decade
L = loyalty2010-2020
Loyalty Decade
Skilled-user Performance
Error-Avoidance
More users, bigger cost impact
• Guest Checkout• One Click Purchase• User Ratings
Investment would increase further
• Competition• Return on Investment• Changing Audience• Internationalization
Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition
Eurostar
• Usability & IA• New booking features• New Brand Identity• Market-leading experience• Six-month timeframe
Metric: Online sales (12 mnths)
Before: £110 million/year
After: £136 million/year
Ratio: 124%Improvement: 24%
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Eurostar
• Issues resolved:– Error Messages– Confusing Language– Confirmation Pages– User Accounts– HTML Issues
Metric: Online sales (12 mnths)
Before: £110 million/year
After: £136 million/year
Ratio: 124%Improvement: 24%
Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition
100+ Issues
70+ Issues
High-fidelity
Adobe Kuler
• Color Theme Sharing Site• Comment Engagement low• Unconventional Display
Metric: # of Comments
Before: 6 / day (Average)
After: 37 / day (Average)
Ratio: 617%Improvement: 517%
Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition
Familiarity does not necessarily breed contempt in Web design. Unless a feature offers entirely new functionality or much improved intuitive use, it is often better to stick with a conventional delivery.
,,,,
Source: Nielsen, J., Berger, J.M., Gilutz, S., Whitenton, K. (2012) Return on Investment (ROI) for Usability 4th Edition
In closing…
Remote Usability Testing:• http://www.optimalworkshop.com/
chalkmark.htm• http://ethn.io/• http://www.feedbackarmy.com/• http://fivesecondtest.com/• http://www.keynote.com/• http://www.loop11.com/• http://www.trymyui.com/• http://usabilitytools.com/• http://www.usabilla.com/• http://www.userlytics.com/• http://www.usertesting.com/
Offline Usability Testing:• http://silverbackapp.com/• http://www.techsmith.com/
camtasia.html• http://www.techsmith.com/
morae.html
A/B Testing:• http://www.adobe.com/africa/solutio
ns/testing-targeting.html
• http://useartisan.com/• http://www.globalmaxer.com/• http://optimizely.com/• https://vwo.com/
Source: http://uxpin.com/upload/ux-design-for-startups-marcin-treder.pdf
Tools, tools, tools…
ClickTracking:• http://www.labsmedia.com/clickheat
/index.html
• http://www.clicktale.com/• http://www.crazyegg.com/• http://usabilitytools.com/features-be
nefits/click-tracking/#tool-description
• http://www.userzoom.com/
Eye Tracking:• http://eyetrackshop.com/
Session Recording:• http://www.userzoom.com/• http://www.ghostrec.com/• http://mouseflow.com/• http://www.openhallway.com/• http://www.tealeaf.com/• http://usabilitytools.com/features-be
nefits/visitor-recording/#tool-description
• http://userreplay.co.uk/
Source: http://uxpin.com/upload/ux-design-for-startups-marcin-treder.pdf
Tools, tools, tools… (more)
Keys to successto help you with your UX measurements
• Make Data Come Alive • Don’t Wait to be Asked to Measure • Measurement is Less Expensive Than You Think • Plan Early • Benchmark Your Products • Explore Your Data • Speak the Language of Business • Show Your Confidence • Don’t Misuse Metrics• Simplify Your Presentation
10x
Source: Tullis, T. & Albert, B. (2008). Measuring the User Experience, 2nd Edition
Image: http://fr.wikipedia.org/wiki/M%C3%A8tre
