© simeon keates 2008 usability with project 15/04/09 susanne frennert

© Simeon Keates 2008

Usability with Project15/04/09Susanne Frennert


Cost-justifying usability


The need to cost-justify

Not all companies (and managers) appreciate the benefits of usability They may cite other factors as more important Examples:

• Tight deadlines• Functionality already developed• Limited money/resources


Cost-justifying usability

However, these are often “false economies” Examples:

• Deadlines:• Releasing the “wrong” product on time is as bad (or worse) as releasing the

“right” product late• Existing functionality:• Existing functionality should have nothing to fear from usability, if it is the

“right” functionality• Limited resources:• Putting the “right” resource in at the “right” time can make the overall project

more efficient


Good usability are good business

Reduced development and software maintenance costs Increased product sales, market share, and revenue Improve customer satisfaction and loyalty Increased site visits and conversion Decrease customer support costs Improve employee productivity Improved brand perception and media coverage


Reduced development and Software Maintenance CostsIntegration of usability testing can contribute to decreased development costs by

Basing feature decisions on solid user testing and evidence decreases the possibility of serious problems later in the process

Many costs are associated with maintenance deal with unmet, or unanticipated, user requirements

“An average of 48% of application code is devoted to the user interface

and 50% of the development time required for the entire application is

devoted to the user interface portion. Given this estimate, incorporating

usability into the design of the UI that customers and end-users are

interacting with is critical.” (Weinshenck 2005)


Increased Product Sale, Market Share, and Revenue

On e-commerce or software as a service site, many potential product sale are lost due to poor site usability.

A 2001 study found that 5% of online shoppers typically fail to find the product or service they are looking for, with as many as 65% of online shopping visits resulting in failure (Souza 2001; Mayhew & Tremaine 2005)

Usability can be also used to increase advertising revenue; a revamp of Yahoo! Overture’s Search marketing tool found a 9% increase in click-through rate after only two weeks and “phenomenal” customer response (Cooper 2008)


Increased Site Visits and Conversions

Usability can also increase the frequency with which users visit the site, contributing directly to the overall market share and customer base

eBay redesign their “Sell your Item” form, which had a high rate of drop-off for first time users, based on their user experience consulting and usability testing. A predicted 1% improvement in the drop off rate contributed to a revenue increase of $1.2 million a year (Braun 2002; Wilson & Rosenbaum 2005)


Improve Customer satisfaction and Loyalty

A product that is easy to use and fun to use creates satisfaction and loyalty in its users

One early study of e-commerce trust found that good navigation and presentation were essential in creating trust, and one of the authors found a link between quality in design and trust in the channel and the company’s brand (Karat & Lund 2005)


Decreases customer support costs

The average call to a software development center is estimated to cost between $12 and $28 per call, and can cost up to $250

(Karat & Lund 2005; Mauro 2005)

“A leading manufacturer of printers released a product that had a serious usability problem in the installment and operation of the printer driver software. This problem was so difficult to solve that more than 50% of the first 100,000 users called the customer service line. The cost was nearly $0.5 million per month. The total cost of replacing the printer driver was $900,000. It was clear upon analysis that the problem could have been identified and corrected at a fraction of the cost if the product had been subject to even the simplest usability testing” (Mauro 2005)


Improve Employee Productivity

Supporting improved, enjoyable-to-use products can increase employee satisfaction and improve the quality of customer service, as well as reducing re-work and “downtime”. These factors contribute to overall improved job satisfaction, which can lead, long term, to improve employee retention (Robb & Pfefer 2003)

“After Oracle changed the navigation structure on its database manager, database administrators were able to perform their duties 20% faster” (Black 2003)


Brand perception and media Coverage

Products with great user experience are recognized for it. Users talk to each other about products they like, increasing sales due to world-of-mouth.

Positive interactions can affect the brand's “halo”.

“Apple’s iPhone, while accounting for only a small blip in the worldwide mobile phone sale, had the most media coverage of any product launched in history, and will likely have long-term effects on the mobile business” (Maurio 2008)


Rosson and Carroll “Usability Engineering”

“Cost-benefit analysis of usability activities contributes to more systematic usability engineering …”

“… BUT benefits are difficult to quantify, so estimates will often be overly conservative.”

Issues: Benefits (e.g. customer satisfaction) are harder to quantify and predict

accurately Costs are very concrete and easy to identify Thus, can be difficult to justify usability…


Typical sources of usability “costs” (also Rosson and Carroll’s usability approach - 1)

Development of requirements scenarios Validation/refinements of scenarios with users and customers Development of basic-level task scenarios Refinement of design scenarios with development team and customers Development of information model Review with team members Development of paper prototypes Walk-throughs with users of paper prototypes …


Typical sources of usability “costs” (also Rosson and Carroll’s usability approach - 2)

… Analysis of transcripts/report preparation Development of interaction model Review with team members Development of running prototypes Formative evaluation Analysis of transcripts/report preparation Detailed design and prototype-driven iteration of previous three steps


Typical usability benefits

Fewer downstream design changes Increased sales (and consequently reduced time to profitability) Reduced need for user training Enhanced customer productivity Reduced resources spent on customer support Increased loyalty in customer base (repeat and referral sales)


A more scientific approach

Mantei and Teorey(*) examined the cost/benefit analysis in 1988

We will look at their calculations

But first we need to examine their methodology

NOTES:• Costs are based on 1988 prices and techniques• Not all costs are required for every project – choose which ones carefully• Costs are indicative, not definitive• Assumes 32,000 lines of code to be used by 250 employees


Typical development stages in a prototyping lifecycle

Feasibility study Requirements definition Global design Prototype construction User evaluation System implementation Testing Update and maintenance


Typical development stages in a prototyping lifecycle

Feasibility study Requirements definition Global design Prototype construction User evaluation System implementation Testing Update and maintenance

Page 19


Typical development stages in a Human Factors software lifecycle

Market analysis Feasibility study Requirements definition Product acceptance analysis Task analysis Global design Prototype construction User testing and evaluation System implementation Product testing User testing Update and maintenance Product survey

Page 20


A breakdown of tangible costs

Cost type Cost (1988 levels)

Salaried employee (SE) Fixed ($40/hr)

Hourly employee (HE) Fixed ($15/hr)

External contractor (EC) Fixed ($60/hr)

Consultant (CN) Variable

Equipment and supplies (ES) Variable

Software purchase (SP) Variable


Further assumptions

40 hours per week 4.3 weeks per month =>172 hours per working month (WM)

Examples: $6880/WM for a salaried employee (at $40/hr)

• i.e. 172 * 40

$10,320/WM for an external contractor • i.e. 172 * 60


Costs added by Human Factors stages

1 – Cost of running focus groups 2 – Cost of building product mock-ups 3 – Expense of the initial design of a prototype 4 – Expense of making a prototyping design change 5 – Expense of purchasing the prototyping software 6 – Cost of running user studies 7 – Cost of creating (or renting) a user study environment (laboratory) 8 – Cost of conducting the user survey


Cost 1 – Focus groups cost breakdown

Time cost of the individuals involved + small equipment cost Individuals involved:

• Participants • Moderator/facilitator• Video-taping/recording personnel• Any other observers/data analysts

Page 24

~3 people

~10 people

1 person


Focus groups cost breakdown

Focus groups typically take 3 hours to run + 1 day to set-up and 1 day to dismantle

Minimum of 2 days to analyse the data

Moderator: Time for focus group + analysis Support staff: Time for set-up + focus group + dismantling Participants: Time for focus group [+ any preparation time]

A complete focus group analysis (3 consecutive groups) takes ~2 weeks

Used for Market Analysis and Product Acceptance Analysis


Focus groups cost breakdown

Estimated costs for conducting focus groups

Type of expense Category Amount

A Cost of operating 3 internal focus groups

30 participants, 10 per focus group (3 hrs each)

SE $3,600

Group moderator (25 hrs) CN $1,500

3 support staff (25 hrs each) HE $1,125

Videotape ES $60

Total $6,285

B Cost of contracting3 external focus groups

Fee charged by agency for complete study ( 3 focus groups and analysis for 2 weeks)

CN $10,000

Page 26


Cost 2 – Estimating product mock-up costs

Building a product mock-up involves constructing a false UI scenario in software and video-taping the scenario

Script has to be written Someone has to execute the scenario Videotape needs to be professional

Videotape mock-up is used in Product Acceptance Analysis• To focus groups to initiate discussions• To target users who are then asked to complete questionnaires about the

use of the product and their response to it

Can also be used in marketing


Estimating product mock-up costs

Costs incurred in building a product mock-up of the proposed software system


Preparation of mock-up scenario (40 hrs) SE $1,600

Videotaping sessions (20 hrs) SE $800

Splicing/integration of scenarios (20 hrs) SE $800

Equipment rental for splicing, etc. ES $500

Videotape ES $60

Total $3,760


Estimating product mock-up costs

The techniques just described are somewhat out of date More usually accomplished via early development cycle prototypes

• e.g. alpha and beta versions• Flash movies, etc.

Q - What to do if no software written? A – Wizard of Oz!


Wizard of Oz

Page 30

Do Action A

{Receive response to Action A}

Do Action B, etc.


Wizard of Oz (The Wizard unmasked!)

Page 31

Do Action A

{Receive response to Action A}

Do Action B, etc.

Create response to Action A

Create response to Action B, etc.


Wizard of Oz explained

User interacts with a UI mock-up only There is no significant software back-end

Remote user (the Wizard) “pretends to be the system” Wizard creates the response based on expected system performance

Still needs a script and plenty of preparation• Imagine if the Wizard does not know a response!


Cost 3 – Estimating user survey costs

Used in the Product Acceptance stage to assess users’ responses to the product mock-up

Also used in Product Survey (after launch) to:• Determine the difficulties users have with the working system• Examine the tasks the system is being used for• Gather suggestions for changes to system

For a user population of 250 employees Typically half (125) would receive a user survey A typical survey is 4 pages in length… … and takes 30 minutes to complete


Estimating user survey costs

Cost breakdown for running a user survey for the software product being tested


Development of questionnaire (40 hrs) SE $1,600

Pilot testing of questionnaire (40 hrs) SE $1,600

Distributing and collecting survey (20 hrs) HE $300

Coding and entering data (20 hrs) HE $300

Analysing the results of the survey (40 hrs) SE $1,600

Cost of time “lost” in filling out survey (40 hrs) SE $1,600

Computer time ES $100

Supplies and duplicating costs ES $100

Total $7,200


Cost 4 – Estimating initial prototype building costs

Cost breakdown for building a prototype does NOT include the design time • Only the building time

Typically 4 weeks

Most prototyping systems involve 2 stages:• Stage 1 – specify the connections between the screen displays• Stage 2 – design each individual screen layout

Alternatively (and more modern)• Stage 1 – specify the states between user interactions• Stage 2 – design the individual states and the alterations that take place

because of user actions


Estimating initial prototype building costs

As the UI grows more complex, time required to build the prototype also increases

Cost of building a prototype is:

C = S × ( a + b D)

Where C = Cost S = Number of states D = Average number of new details per state a = Constant reflecting the cost of building a single state b = Constant reflecting the cost of adding a single detail


Estimating initial prototype building costs

Costs incurred in the initial design of a prototype of the proposed software system (the prior development of a global design is assumed)


Specifications of the screen transitions (80 hrs) SE $3,200

Design of the individual screen layouts (80 hrs) SE $3,200

Total $6,400


Cost 5 – Estimating costs for design changes to original prototype

Once the prototype is built, user studies will uncover problems with it• Difficulties learning and using the system

Design change suggestions will be made … … incorporated into the design … … and evaluated again, etc. … … until the number and severity of problems are “acceptable”

The initial user studies will usually find the most and most major issues Later changes should be minimal

• Especially if the prototype is close to the final product design …• … which it should be if task analyses, user surveys, etc. have been used• Also, design changes should simply be updates of parts of the prototype …• … and not a complete re-design


Estimating costs for design changes to original prototype

The initial user studies will usually find the most and most major issues Later changes should be minimal

Especially if the prototype is close to the final product design … … which it should be if task analyses, user surveys, etc. have been

used

Also, design changes should simply be updates of parts of the prototype …

… and not a complete re-design



Since changes should be restricted to parts of the UI, should only take ~1 day per change

The number of changes expected and amount of time per change depends upon complexity of the interface

Similar equation to cost of building a prototype



Costs incurred in incorporating a design change to the original prototype of the proposed software system. These costs assume that the design change does not require a complete revision of the screen transitions.


Modification of the screen transitions (4 hrs) SE $160

Re-design of the individual screen layouts (4 hrs) SE $160

Total $320


Cost 6 – Estimating the cost of purchasing the prototyping software purchase

Actual purchase cost Also time spent deciding which one … … and then learning it

1988 prices - $10,000 for the software ($2,500 to $15,000+) 2008 prices - $600 per user (AXURE)


Estimating the cost of purchasing the prototyping software purchase

Page 43

Costs incurred in incorporating a design change to the original prototype of the proposed software system. These costs assume that the design change does not require a complete revision of the screen transitions.


Time spent reviewing potential packages (1 month) SE $6,080

Purchase cost of package SP $10,000

Total $16,080

+ Time spent learning package


Cost 7 – Estimating costs of performing user studies

Preparation time can be substantial• Example: preparing a manual for a completely new UI• Manual need not be complete, but it needs to be sufficient and pilot-tested• Example: preparing the study protocol• Protocol needs to be complete and pilot tested

Costs of individual user studies are often independent of complexity of UI …

… however, the number of user studies increases with complexity

3 types of user study in the Human Factors software lifecycle…


Estimating costs of performing user studies

In Task Analysis: Users are asked to perform the types of tasks the system should

support Aim is to build a model of how users approach the task

In User Testing and Evaluation and in final User Testing: More conventional user studies Conducted on prototypes or final system Final system evaluation is always needed

• May be significant changes from the prototype behaviour


Estimating costs of performing user studies

Costs incurred in conducting a single user study on 5 participants


Development of user directions [i.e. protocol] (40 hrs) SE $1,600

Pilot testing of directions (20 hrs) SE $800

Re-designing user directions (20 hrs) SE $800

Running experiment (40 hrs) SE $1,600

Analysing results of lab study (40 hrs) SE $1,600

Videotape ES $120

Cost of users in experiment (20 hrs) SE $800

Total $7,320


Cost 8 – Estimating costs of construction of a user laboratory

A user lab can be a borrowed / rented or permanent office Permanent office is recommended where significant usability activity is

expected User lab should be a mock-up of the environment of use for the

product The lab should allow space for an adjoining observation/recording

room Construction of a lab takes ~5 weeks


Estimating costs of construction of a user laboratory

Cost of establishing a permanent human factors lab. These are mid-level costs. Much more sophisticated labs can be built.


Time spent laying out lab design and selecting lab equipment (160 hrs)

SE $6,400

Cost of carpenter and electrician (20 hrs) EC $1,200

Cost of cameras, VCRs, one-way mirror ES $10,800

Total $17,600


Estimating costs – a summary

Lifecycle stage Cost item Total cost

Market analysis Focus group (set of 3) $6.285

Product acceptance analysis Focus group (set of 3) $6.285

Product mock-up $3,760

User survey $7,200

Task analysis User study $7,320

Lab construction $17,600

Prototype construction Initial design $6,400

Design change (20 @ $320) $6,400

UIMS system $16,080

User testing and evaluation User study (4 @ $7,320) $29,280

User survey $7,200

User testing User study $7,320

Product survey User survey $7,200

Total $128,330


Estimating times – a summaryLifecycle stage Cost item Total WM Dev time

Market analysis Focus group (set of 3) 1.11 6 wks

Product acceptance analysis

Focus group (set of 3) 1.11 6 wks

Product mock-up 0.47 2 wks

User survey 1.16 4 wks

Task analysis User study 1.05 4.3 wks

Lab construction 1.05 5 wks

Prototype construction Initial design 0.93 4 wks

Design change (20 @ .05 WM) 1.00 4 wks

UIMS system 1.00 4.3 wks

User testing and evaluation

User study (4 @ 1.05 WM) 4.20 17.2 wks

User survey 1.16 4 wks

User testing User study 1.05 4.3 wks

Product survey User survey 1.16 4 wks

Total 16.45 WM 69.1 wks


Common tangible benefits of Human Factors design

Direct benefits can be calculated by making several valid assumptions about the improvements to the UI, specifically:

(1) A reduction in user learning times

(2) A reduction in user errors

(3) A reduction in the cost of maintaining the system

Remember: for 32,000 line program for 250 employees…


Intangible costs of Human Factors software lifecycle

1 - The selection of non-critical design decisions for user studies 2 - The establishment of too high a level of usability 3 - Falling into the trap of overdesign 4 - Communication problems between Human Factors specialists and

software designers


Selection of non-critical design decisions for user studies

Not all design decisions affect the overall “quality” and “acceptability” of the final UI

How do you determine which ones are worth investigating?• Examples: Arial or Helvetica? 11 pt or 12 pt? Icon names?

Most usability experts rely on experience and intuition to decide… … but they can be wrong!

Costs: Wasted user study time Discovery of important changes may be put off until later

Solution: Make your user studies flexible enough to allow discovery of new problems


Establishment of too high a level of usability

Some UI performance targets simply cannot be met Example: “This system must be learnable within 1 week” This may appear to be a reasonable target … … but what if the task itself takes >1 week to learn?

Costs: Design and development time chasing unrealistic performance targets

Solution: Set realistic targets This can be informed by your task analysis studies Also, aim for incremental improvements in performance over design

cycles


Falling into the trap of overdesign

Prototyping software (UIMS) can make it very easy to add more features (“bells and whistles”)• Examples: borders, colours, icons, images, etc.

How to know when to stop??? How to know what are valuable additions?

Costs: Designer’s time + implementation time

Solution: Strong management control


Communication problems between Human Factors specialists and software designers

There is a knowledge gap between HF specialists and the coders/developers…

Unless there is a shared common language and understanding, it is difficult to communicate effectively

Costs: Time lost in establishing a common language/understanding Time lost developing the wrong thing (or the thing wrongly) Time lost designing solutions that simply cannot be implemented

Solutions: HF specialists learns to develop own UI Better corporate culture of communication


Intangible benefits from Human Factors software lifecycle

1 - Adoption of features that save time 2 - Avoiding system sabotage problems 3 - Enhancing the ability to solve conceptual problems using the

software system


Adoption of features that save time

Feature adoption by users is reduced when the system is so complex that users give up trying to learn the advanced features• i.e. they satisfice

Users will typically adopt the least complex system that offers them the required functionality• Even if it is less efficient

The Product Acceptance Analysis should identify unnecessary features

The User Tests should reduce feature complexity


Avoiding system sabotage problems

Being asked to use a system that is inappropriate, difficult or inadequate can lead to employee frustration

This frustration can lead to employees “taking it out on the system”• Example: entering incorrect or incomplete data• Example: reporting false system failures

This is referred to as “system sabotage”

Focus groups in Market Analysis and Product Acceptance stages are designed to address this issue• They test the receptivity of the target users to the new system

Also, the Task Analysis ensures that the final product matches the needs of the users as closely as possible


Enhancing the ability to solve conceptual problems using the software system

Increasing the user’s cognitive load to use the system … … decreases the user’s cognitive capacity for “solving the problem”

So although the problem may be “solved” … … there may have been a “better” solution

User Testing is designed to remove complexities from the system … … and thus support users’ creative problem solving


Recommendations for Human Factors inclusion

The HF lifecycle stages and the type of cost reduction that they are most likely to effect

Cost reduction item Related lifecycle stage

Increased system adoption Market analysis

Product acceptance analysis

User testing and evaluation

Reduced training costs Task analysis

User testing

Reduced user errors Task analysis

User testing

Transfer of design changes to an earlier stage in the lifecycle

Prototype construction

User testing (on prototype)

Product survey (next re-design)


Cost of running User Tests vs. size of user population

Page 62

Size of user population

$

Break-even point

Benefits of testing

Cost of testing

User testing not cost-effective

User testing cost-effective


Summary of cost-benefit analysis

For any large user group, the benefits will usually outstrip the costs … …often by a significant margin

For smaller user groups, the case is less clear-cut … … so either decrease costs …

• e.g. through “discount” usability methods

… or examine the intangible benefits more closely


Designing for mobile devices


5 products killed by the mobile phone (according to Wired.com)

1 – The PDA 2 – The camera 3 – The Ultra Mobile PC 4 – The telephone 5 – The MP3 player 6 – Books 7 – Handheld games consoles


Designing for Mobile devices(Jakob Nielsen's Alertbox, February 17, 2009)

Abysmal success ratesUsers failed more often than they succeeded when using their mobiles to perform tasks on websites

Download timesMost pages take far too long to load, particularly on non-3G phones. But even the highest-end phones deliver much slower browsing than a desktop computer. As a result, users are reluctant to request additional pages and they easily give up.

Search Dominance

Mobile users are more likely to want information to help them at that location or time, such as finding directions or finding out what's going on nearby.


Mobile Internet Motivations and behaviour (Carol Taylor, Judith Ramey, Motricity & University of Washington)

Motivation 1. Awareness: The desire to stay current,

to keep oneself informed in general. Examples:

scanning email and checking news sites.

Motivation 2. Time Management: The desire to be

efficient, to manage projects, or get things done.

Examples: looking up an address; checking traffic

maps; looking for supplies/ jobs/ roommates; getting

instructions for a class assignment.

Motivation 3. Curiosity: The interest in an unfamiliar

topic, often based on a tip or chance encounter.

Examples: looking up information about a country of

interest or information to settle a friendly bet in a bar.


Mobile Internet Motivations and behaviour (Carol Taylor, Judith Ramey, Motricity & University of Washington (CHI 2009))

Motivation 4. Diversion: The desire to kill time or

alleviate boredom. Examples: browsing favorite sites;

checking social networking sites.

Motivation 5. Social Connection: The desire to

engage with other people. Examples: arranging to get

together; sending email; posting to social networking

sites; seeking information as a group.

Motivation 6. Social Avoidance: The desire to

separate oneself from others, to appear occupied so as

not to be bothered. Examples: using cell phone activity

as a “cover” to prevent others from talking to you.


Designing for Mobile devices

Excellent sets of slides at from MobileHCI 2008

http://albrecht-schmidt.blogspot.com/2008/09/mobilehci-2008-tutorial.html

Suggested reading: Scott Mackenzie’s presentation “Text input for mobile devices”…


Exercise


Exercise

Conclude your user trials

If you want feedback on your group presentation – then prepare slides for Friday 4th of December• NOTE – this is optional, but recommended

© simeon keates 2008 usability with project 15/04/09 susanne frennert

Documents