exploring usability testing for mobile and web technologies

TJ AM Tutorial

10/14/2014 8:30:00 AM

"Exploring Usability Testing for Mobile

and Web Technologies"

Presented by:

Rob Sabourin

AmiBug.com

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Rob Sabourin

AmiBug.com Rob Sabourin, P. Eng., has more than thirty years of management experience leading teams of software development professionals. A well-respected member of the software engineering community, Rob has managed, trained, mentored, and coached hundreds of top professionals in the field. He frequently speaks at conferences and writes on software engineering, SQA, testing, management, and internationalization. Rob wrote I am a Bug!, the popular software testing children's book; works as an adjunct professor of software engineering at McGill University; and serves as the principle consultant (and president/janitor) of AmiBug.Com, Inc. Contact Rob [email protected].

© 2013 Robert Sabourin 1

© 2013 Robert Sabourin EUT v1.1 1

Exploring Usability Testing

Robert Sabourin

President

AmiBug.Com, Inc.

Montreal, Canada

[email protected]

WELCOME




• Robert Sabourin ,

Software Evangelist

• President

• AmiBug.Com Inc.

• Montreal, Quebec,

Canada

• [email protected]

• www.amibugshare.com





•Net Promoter Score

•System Usability Scale

•Exploratory Testing

•Whiteboarding

•Usability Heuristics





•Forms and Workflows

•Usability Reviews

•Do It Yourself Usability Testing

•Some Mobility Usability Concerns

•Some Accessibility Concerns


NON-FUNCTIONAL TESTING




Non-Functional Testing

Quality Factors

Attributes

Characteristics

Reliability

Other “-ilities”


Non-Functional Testing Challenges

Goals

• Elicit

• Articulate

• Quantify

Subjective

• Needs or wants?

• Relative or absolute?

• What is good enough?

Tests

• Tricky to orchestrate

• Difficult to interpret results

• Challenging to baseline or regress

8© 2013 Robert Sabourin EUT v1.1


Non-Functional Agile Challenges

Time

• Analysis

• Model

• Prepare

• Run

• Interpret

Change

• Baselines

• Goals

Software

• Continuous integration

• Incomplete product builds



Principles

• Seek to understand why attributes are valued by stakeholders

• Seek to discover relationship between non-functional test and value

• Use multiple tools and techniques

• Use heuristic guides



ISO 9126 Quality Factors

•Attributes of the effort needed for use, and on the individual assessment of such use, by a set of users

•Learnability

•Understandibility

•Operability

Usability


Quality Factors



Product Backlog Constraints

Tom Gilb, Competitive Engineering, suggests defining

Scale: "What is measured"

Meter: "How to measure (method)"

Target: "Level we're aiming for. Success"

Constraint: "Level we're seeking to avoid. Failure"

Benchmark: "Where we are today"





Non-Functional Experimentation• Iterative

• Implement trial

• Measures result

Timebox non-functional tests

• Pause

• Review findingsAfter timebox

• Investigate more

• Refactor goal

• Move on to something elseNext steps


Experimenting

Conjecture

ConfirmationRefutation



Non-Functional Exploratory Testing

Usability

Pair with operator

Pair with domain experts

Pair with human factors guru


NET PROMOTER SCORE




Net Promoter Score


Net Promoter Score



Net Promoter Score

•11 values

•0 not at all likely

•10 extremely likely

Response Scale


Net Promoter Score

• respondents giving a 9 or 10 score

Promoters

• respondents giving a 7 or 8 score

Passives

• respondents giving a 0 to 6 score

Detractors



Net Promoter Score

•The difference between the percentage of Promoters and Detractors

• Is not expressed as a percentage

• Is an absolute number lying between -100 and +100

•uTest suggests “any positive number is good” (ref: TK Maxx Mobile Usability Testing Results Report)

NPS


Net Promoter Score




SYSTEM USABILITY SCALE




System Usability Scale

•Measures usability in terms of effectiveness, efficiency and satisfaction

• SUS can be read as a percentage

• SUS can be used as a benchmarking mechanism

• SUS can be used to compare the current system to competitors’ systems

• SUS can be used to compare the current system to future versions

SUS - System Usability Scale






1. I think that I would like to use this system frequently2. I found the system unnecessarily complex3. I thought the system was easy to use4. I think that I would need the support of a technical person to be able to use this system

5. I found the various functions in this system were well integrated

6. I thought there was too much inconsistency in this system

7. I would imagine that most people would learn to use this system very quickly

8. I found the system very cumbersome to use9. I felt very confident using the system10. I needed to learn a lot of things before I could get going with this system


SUS Questions



SUS Responses



• For odd items: subtract one from the user response.

• For even-numbered items: subtract the user responses from 5

• This scales all values from 0 to 4 (with four being the most positive response).

• Add up the converted responses for each user and multiply that total by 2.5. This converts the range of possible values from 0 to 100 instead of from 0 to 40.


Scoring SUS



Scoring SUS


EXPLORATORY TESTING



Exploratory Testing

© 2013 Robert Sabourin EUT v1.1

Learning

DesignExecution

34


Charter Statement

• Statement of mission

• Ties to purpose

• Focuses work

• Confirms understanding

• Delineates scope

• Analogy to test story


Charter Statement

• Short, terse

• To the point

• Inclusions '

• Exclusions '

• Limits '



eBay Usability Charters





Pairing Testers With …

Non Functional Testing

•Usability Gurus

•SMEs

•Human Factors Experts


WHITEBOARDING




Prototype Styles

• Paper and pencil• Comic books style story boards• Wire frame models• Mock ups• Prototypes• Site maps• User flow• Wire flow• Story boards• Wire frame• Page description diagram• Functional specification


Whiteboarding



Whiteboarding


Whiteboarding



Whiteboarding

What is Whiteboarding?

Collaboration

Coordination

Conceptualization


Tony Buzan

Mind Maps



Whiteboarding


Whiteboarding

User Mind Map



Buying a Book

Usage Scenarios


Whiteboarding

Story Board



Wrap-O-Matic

Usage Scenarios


Scenario Based Testing



Buying a Book

Usage Scenarios



Slide 56



Slide 57



Social Networking

Usage Scenarios




USABILITY HEURISTICS



Jakob Nielsen

• Usability heuristics

• Rules of thumb

• General principles of user interface design

• www.nngroup.com



Usability Heuristics

• Visibility of system status

– Always keep users informed

– Provide appropriate feedback

– Respond in a reasonable timeframe



• Match between system and the real world

– Speak the users' language

– Use familiar concepts

– Follow a natural order




• User control and freedom

– Provide means to exit from unwanted states

– Provide undo and redo capabilities



• Consistency and standards

– Same words mean same thing in different contexts

– Follow environment conventions




• Error prevention

– Prevents problems from occurring

– Eliminate error-prone conditions

– Have a confirmation option



• Recognition rather than recall

– Making objects visible

– Make instructions available




• Flexibility and efficiency of use

– Accelerators for the expert user

– Make common options adaptable



• Aesthetic and minimalist design

– Dialogues should contain relevant information




• Help users recognize, diagnose, and recover from errors

– Use plain language for error messages

– Constructively suggest solutions



• Help and documentation

– Easy to search

– Focused on the user's task

– Offer concrete steps to be carried out

– Short and to the point



USABILITY SEVERITY LEVELS



Usability Severity



ISO 9421



Usability ISO 9421

•the extent to which the intended goals of use are achieved

Effectiveness

•the resources that have been expended to achieve the intended goals

Efficiency

•the extent to which the user finds the use of the product acceptable

Satisfaction

•can user be harmed

Safety



Usability ISO 9421

•the extent to which the system is complete in design and meets its desired goals

Completeness

•Is the capability of a software product to enable the user to learn how to use it

Learn-ability

•the extent to which the product fits into normal course of daily life

Convenience

•Confusing or vague usability. Misleading information that might also result in providing no usability at all

Ambiguity


FORMS AND WORKFLOWS




Form Design Principles

• Minimize pain• Illuminate path• Consider context• Ensure consistent communication


Luke Wroblewski

Web Form Design

Eye Tracking



Form Usability

• Number and location of errors• Severity of errors• Completion rates• Time to complete forms• Satisfaction scores• Subjective comments


Awkward Form Path



Clear Form Path


Form Usability

• Eye tracking– What people looked at

– Number of eye fixations

– Length of eye fixations



Form Usability Checklist

�Any unnecessary questions�Succinct questions�Sincere voice�Natural language�Logical groups of information�Structure form as a conversation�Ask optional questions separately


Form Usability Checklist

�Name the form�Name sections of the form�Clear scan line�Spacing between sections�Minimal distractions�Start page guidance for long forms�Indicate progress



Progress Indication


Input Validation

�Suggest valid answers�Illustrate valid format�Validate syntax after user enters data�Indicate limits or ranges of inputs�Provide smart defaults�Hide unneeded controls



Error Messages

�Provide clear error messages�Indicate if error blocks completion�Guide users to resolve error�Provide visual emphasis of error�Use red text and icons for errors�Indicate successful completion


DO IT YOURSELF USABILITY TESTING




Do it yourself


Do it yourself

A morning a month

Get insights from users

Watch real users



Do it yourself

Recruit diversely

Find important problems

Improve design


Do it yourself

Three testers

Test on site

Observe shared screens



Do it yourself

Observers take notes

User questions recorded

Action decided at lunch


SOME MOBILE USABILITY CONCERNS




Mobile Usability

•Small Screens

•Limited bandwidth

•Fingers

•No Flash Like GUI

Difficulties


Mobile Usability

•Bad first impression

•Operator errors

•Negative feedback / reviews

•Competitor fills gap

Risks



Mobile Usability

• Important stuff on top

• Easy to find stuff

• Single column layout

• Minimize navigation

• Light data

Guidelines


Mobile Usability

• Finger taps

• Avoid fancy web gui technologies

• Minimize forms

• Minimize feature sets

• Separate mobile web resources

• Fast

Guidelines



Mobile Usability

•In the lab

•Beta

•Crowd source

Mobile Usability Test


Mobile Usability

• Vary and control

• Technologically

• OS

• Browser

• Device

• Carrier

• Bandwidth

Mobile Usability Test



Mobile Usability

•Geographically

• Continent

• Country

•City

• Language

• Locale

Vary and control


Mobile Usability

•Demographically

•Age

•Gender

• Education

•Employment

• Industry

•Computer savvy

•Domain expertise

Vary and control



SOME ACCESSIBILITY CONCERNS



Accessibility

Does application support

•Special needs of end user

•Disabilities of end user

•Older end users

•End users in rural areas

•End users in developing countries



Accessibility

• Difficult to see

• Difficult to hear

• Difficult to interpret text

• Difficult to point to object

• Difficult to use keyboard

• Difficult to use mouse

Some Examples


Accessibility

Tools support http://www.w3.org/WAI/ER/tools/complete

• Bobby (version 5.3)Watchfire, 14 March 2005Description: Watchfire® Bobby™ 5.0 is a web accessibility desktop testing tool designed to help expose barriers to accessibility and encourage compliance with existing accessibility guidelines, including Section 508 of the U.S. Rehabilitation Act and the W3C's Web Content Accessibility Guidelines (WCAG). Bobby spiders through a website and tests to see if it meets accessibility requirements, including readability by screen readers, the provision of text equivalents for all images, animated elements and audio and video displays. During a scan, Bobby checks HTML against select accessibility guidelines and then reports on the accessibility of each web page.Guidelines: WCAG 1.0, Section 508Automatic checking: Single pages, Page groups, Restricted pagesFormats: HTML, XHTML



Accessibility

Accessibility Failures

• http://www.w3.org/standards/webdesign/accessibility

• Taxonomy of example accessibility failures


Accessibility

•http://www.color-blindness.com/coblis-color-blindness-simulator/

• http://www.vischeck.com

Color Blindness Simulator

•http://www.screen-resolution.com/

Resolution Simulator



Bibliography

• Casey, S. (1998), Set Phasers to Stun: And other true tales of design, technology, and humanerror, Agean, Santa Barbara.

• Covey, S. (2000), Living the 7 Habits : The courage to change, Simon & Schuster, New York.

• Crosby, P. B. (1990), Let's Talk Quality: 96 Questions You Always Wanted to Ask Phil Crosby, Plume, New York.

• Crosby, P. B. (1988), The Eternally Successful Organization, Penguin Group, New York.

• Derman, E. (2011), Models.Behaving.Badly: Why confusing illusion with reality can lead to disaster, on wall street and in life, Free Press, New York.

• Dustin, E. (2002), Quality Web Systems : Performance, security, and usability, Addison Wesley, Boston.

• Fine, M. R. (2002), Beta Testing for Better Software, Wiley Technology Publishing, New York.

• Gawande, A. (2009), The Checklist Manifesto, Metropolitan Books, New York.

• Gigerenzer, G. & Todd, P. M. (1999), Simple Heuristics That Make Us Smart, Oxford University Press, Oxford, UK.

• Gilb, T. (2005), Competitive Engineering: A Handbook For Systems Engineering, Requirements Engineering, and Software Engineering Using Planguage, Butterworth-Heinemann, Amsterdam, Netherlands.


Bibliography

• Gilb, T. & Graham, D. (1993), Software Inspection, Addison-Wesley, Harlow, England.

• Gilb, T. & Weinberg, G. (1977), Humanized Input: Techniques for Reliable Keyed Input, Winthrop Publishers, Inc., Wellesley, MA.

• Guaspari, J. (1985), I Know It When I See It, Amacon, New York.• indle, T. (1998), Interviewing Skills, Fenn Publishing Company Ltd., Bolton, ON.

• Jonassen, D. H.; Tessmer, M. & Hannum, W. H. (1999), Task Analysis Methods for Instructional Design, Lawrence Erlbaum Associates, Publishers, Mahwah, New Jersey.

• Kaner, S.; Lind, L.; Toldi, C.; Fisk, S. & Berger, D. (1996), Facilitator's Guide to Participatory Decision-Making, New Society Publishers, Gabiola Island, BC.

• Kluger, J. (2009), Simplexity : Why simple things become complex (and how complex things can be made simple), Hyperion, New York.

• Krug, S. (2010), Rocket Surgery Made Easy, New Riders, Berkeley, Califonia.

• Krug, S. (2006), Don't Make Me Think, New Riders, Berkeley, California.

• Norman, D. (1993), Things That Make us Smart : Defending human attributes in the age of the machine, Perseus Books, Reading, Mass.

• Norman, D. A. (1999), The Invisible Computer: Why Good Products Can Fail, the Personal Computer Is So Complex, and Information Appliances Are the Solution, The MIT Press, Cambridge, MA.



Bibliography

• Patton, M. Q. (2002), Qualitative Research & Evaluation Methods, Sage Publications, Inc., Thousand Oaks, CA.

• Roam, D. (2011), Blah Blah Blah : What to do when words don't work, Portfolio/Penguin, New York.

• Roam, D. (2009), Unfolding the Napkin : The hands-on method for solving complex problems with simple pictures, Portfolio, New York.

• Roam, D. (2008), The Back of the Napkin: Solving Problems and Selling Ideas with Pictures, Penguin Books, New York.

• Rosson, M. (2002), Usability Engineering : Scenario-based development of human-computer interaction, Academic Press, San Francisco.

• Stephens, M. (2010), Design Driven Testing : Test smarter, not harder, Apress, New York.

• Surowiecki, J. (2005), The Wisdom of Crowds, Anchor Books, New York.

• Ware, C. (2008), Visual Thinking for Design, Morgan Kaufmann Publishers, Burlington, Massachusetts.

• Carrol, J., ed. (1995), Scenario-Based Design: Envisioning work and technology in system development, John Wiley & Sons, New York.


exploring usability testing for mobile and web technologies

Technology

robert sabourin eut

rob sabourin

popular software testing

software engineering

contact rob

competitive engineering

presidentjanitor of

workflowsusability reviewsdo