Usability Research:Unexpected Results

Overview

• User feedback and user performance

• Unexpected results in research/usability

• Small scale research accuracy

Product: College Writing Handbook

How it is tested• Textbook publishers – experts• Computer industry – usability tests

Product: College Writing Handbook

Research Goals• Convince publishers of value of usability testing• Compare visual against traditional version• Help find/fix problems in new handbook• Usability research: new thinking about old products

Delivery of Material

Old way: prose instruction Traits• Traditional Vocabulary• Information listed by bullets• Emphasis – minimal visuals

– “^”– Italics

Delivery of Material

New Way: artwork Traits• Minimal grammar

terminology• Color coding• Simpler language• “adding a medial modifier

to an independent clause”

User Profiles: 12 Participants

• All 18 or 19 years old• First semester, first composition course• Engineering, Business, etc• 6 student from 2 year college

– 4 male, 2 female

• 6 students from 4 year college– 3 male, 3 female

Scenarios

1. “Put complex source into correct MLA style”2. “Identifying non-trivial comma errors”3. Evaluate source acceptability for assignment/audience

• Talk out loud while using textbook• Point at text while reading• Prompted after 5 seconds of silence• Videotaped

Scenario 1: Citation

• Two sources provided• Use both handbooks – alternate handbook used first• Ratings:

– Very useful – Useful – Rarely Useful – Not useful

• Explain the rating

Scenario 2: Punctuation

• Evaluate a paragraph “pregnant with comma errors”– Comma required?– Comma optional?

• Ease of use• Explain the rating

Scenario 3: Acceptability of Source

• Users given a research topic and audience• Users given possible sources

– Acceptable?– Unacceptable?– More information required?

• Ease of use

“Print Quality Bias”

• Prototype vs finished product• Color copies of prototype vs color excerpts• Texts plastic comb bound • Both texts referred to as prototypes

Findings

• Visual product preferred by users

• Verbal product rated slightly more difficult

“Users failed at tasks, but didn’t realize it”

• Ease of use does not equal usability

• Works cited - both prototypes failed users – 12 unsatisfactory work cited entries– Minor omissions: “Press” or “Inc”– Critical omissions: authors, title, edition number, pages

• Punctuation – 11 of 12 students misuse comma

• Source acceptability

Howard 10

Creating a citation

• Positive responses• 12/12 user failure• Problem areas

– Large font, highlighting, underlining

– Users misled – other info required for citation

“In America* it is quite possible to live a cocoon.”

• Correct Response– Comma optional at *– Visual prototype - page 433– Verbal prototype - page 236

• User results– 11/12 gave “required” as response – both books– Visual prototype – 12/12 cited the correct page– Verbal – 12/12 cited incorrect page

Explanation of failures

• “Readers scanned pages for examples that matched mental models”

• “They thought the problem was simple and didn’t look beyond the first solution…”

• Relied on bold headings, skipping paragraphs

Howard 11

Explanation of failures cont’d

• “Visual manual tried to combine too much information in one graphic.”

• “Authors of the manuals didn’t understand their users’ mental models.”

• One text failed: possible delivery problem?

Howard 11

First Simple Solution

• Users appeared to focus on bold headings• Scanned examples• Looked for examples to match pattern of task• Rarely read prose paragraphs

Additional Issues

• Visual is too complex• User comments

– “’tangled up’” or “’messy’”– “’Too busy’”– “Too much effort”– Skipped it

Preferred page – why?

• Users scan for syntax patterns

• “Does not…combine elements into one visual”

Acceptability of Source

• Step-by-step instructions: too simple• Provide context or “’If, Then’” scenarios• Visual book used “stories”

– Pedro, Aaliyah with respective assignments– Both students evaluate the same source– Story shows decision making process– User-centered design? User-experience!

Context

• Both texts made assumptions– Knowledge of corporate authors, reference books, etc– How to determine the type of source

• Fixes– Task Environment– “’How do quote or paraphrase in my text?’”– “’How do I format entry for works cited, reference list,

etc”

Howard 14

Other Results

• Fixes for handbooks – visuals, complexity indicators

• Total client focus can be bad– “I like/want this!”– Focus: task completion AND decision making

• Make users aware of complexity – context– Usability Test/final product– Model problem solving behavior in usability test

Small Scale Research Accuracy

Small Numbers?

• Revised handbook • Usability results vs actual user results?• Task success• Oversimplified results• Extreme results

Confidence Interval

• 95% - by convention• 95% of the time - results fall within planned range• Based on sample size and success rate

– 5 users – large margin for error– 100 users – smaller margin for error

Confidence Interval

• 5 users - 95% of the time, completion of tasks will be between 48% - 100%

Adjustment – Maximum Likelihood Estimate (MLE)

• Successful attempts (x)/total attempts(n)

• x/n = probability of success (p)• x/n = p • 4/5 = .80 or 80%

Adjustment – Jeffreys Method

• Successful attempts (x)/total attempts(n)

• x/n = probability of success (p)• x/n = p• (x+.5)/(n+1)

• 4/5 = .80 or 80%• 4.5/6 = .75 or 75%

Adjustment – Laplace Method

• Successful attempts (x)/total attempts(n)

• x/n = probability of success (p)• x/n = p• (x+1)/(n+2)

• 4/5 = .80 or 80%• 5/7= .714 or 71.4%

Adjustment – Wilson Method

• Successful attempts (x)/total attempts(n)

• x/n = probability of success (p)• x/n = p• (x+2)/(n+4)

• 4/5 = .80 or 80%• 6/9= .667 or 66.7%

Adjustment Review

• 5/5 = 100% - Really?

• 4/5 = .80 80% MLE• 4.5/6 = .75 75%

Jeffreys• 5/7= .714 74.1%

Laplace• 6/9= .667 66.7% Wilson

Keep it Simple

• Sample less than 20, use adjustment method

• www.measuringusability.com/wald

• 4 of 5 users succeed: 71.4%

Lewis and Sauro 2-15

Summary

• Writing Handbook• Unexpected results – usable, but not useful• Adjusting for small samples• http://www.upassoc.org/upa_publications/jus/• www.measuringusability.com/wald