planning and running usability tests

Usability testing

Planning & runningChris Collingridge (@ccollingridge)

22 October 2013

Manchester Metropolitan University 1


1.0 Who are we and who am I?2.0 What type of test?3.0 Fitting it in4.0 Planning and doing


Sage and meWho we are and what we do


Sage – Global

• 6 million customers

• 13,300 employees

• Major offices in UK, Ireland, France, Germany, Spain, USA, Canada, Austrailia, & Brazil

– Small business accounting– Payroll– Customer relationship management

(CRM)– Taxation and accountancy – Electronic payments


Sage – UK

• Only software company in the FTSE 100

• 800,000 UK businesses use Sage

• #1 in small business accounting

• 1 in 4 people in the UK are paid by Sage Payroll


Sage – Manchester

• Software for accountants in practice

• On-premise and online software

• 300,000 sets of company accounts filed using Sage each year

• 200,000 corporate tax submissions

• 520,000 personal tax submissions

– Final accounts production– Corporate and personal taxation– Practice management – Time recording and billing– Accountant/client collaboration


Summary

We’re big


Me

– Degree in economics (obviously!)– Worked in a shop– Decided there must be a career in

computers

…and mainly self-taught 15 years later…

– Senior User Experience Specialist!


Day-to-day

• User research – understanding the problem

• What do people know?• What are they trying to do?• Where do they do things?• What do they value?• What troubles them?

• Interaction design – solving the problem

• Information architecture• User flows• Patterns• Low-level interaction (controls etc.)

• Usability testing – evaluating solutions


What type of test?


Formative vs. Summative

• Summative

“evaluation of a product with representative users and tasks designed to measure the usability (defined as effectiveness, efficiency and satisfaction) of the complete product…the main purpose of a summative test is to evaluate a product through defined measures, rather than diagnosis and correction of specific design problems”

Ref: Usability Body of Knowledge

• Formative

“a type of usability evaluation that helps to "form" the design for a product or service. Formative evaluations involve evaluating a product or service during development, often iteratively, with the goal of detecting and eliminating usability problems.”


Formative vs. Summative

• Summative

Analytics

Customers surveying and feedback

“Voice of customer”

Usability test = large scale, expensive, scientific, resource intensive, low ROI

• Formative

Most of our usability testing is formative

Blended with research

Often conceptual

Usability test = small scale, iterative, pragmatic, high ROI


Fitting it in


Overall process

Plan Prepare Do Analyse Act

This is the usability test bit


Planning


Fact*

96% of #### usability tests are a result of poor planning*

* Entirely made up


Failure

• Unclear goals

• Test not addressing goals

• Method produces unusable results

• Too many variables

• Unrepresentative participants

• Inconsistent moderation

• Inconsistent/incomplete notes/recording

= a whole load of rubbish

Let’s plan!


Planning


My template for planning

Objectives

Goals

Questions

Method

Participants

Practicalities


1 – Objectives

What will you be able to do as a consequence of this test?

The point is always to stimulate valuable action

Example: Redesign the delivery elements of the shopping cart


1 – Objectives

Your objective is not a report


2 – Goals

What will you know about or understand?

Examples: • Know how people enter addresses for other people if

they’re sending gifts• Understand how people react to different default delivery

prices


3 – Questions

What specific questions will you have answers to?

Examples: • When asked to enter the address of a friend or sibling,

where do they get the address from?• Do postcode lookups help people enter addresses other

than their own?• Are people less willing to continue if higher priced delivery

options are the default?


4 – Method

How will you get answers to the questions?

Hint:• Don’t think only about checking something

• Compare one thing to another (but control your variables)

• Evaluate the perceived value of something• Gain contextualise insight into something


4 – Method

Some things to think about when defining your method (1)

• Do you need to be able to see the user?• Do they need to be using their own equipment?

• (PC vs. Apple, desktop vs. laptop, multi-screen, browser, etc.)

• How much time will they have?• How much time do you have?• What equipment/software will you be able to use?• How much help might they need to access the software?


4 – Method

Some things to think about when defining your method (2)

• Where can you run it?• What do you need to be able to measure?

• Time*, success rate, errors, comments?• How will you avoid order effects?• Are you able to access/build software to support the

tasks?

* Using “talk aloud” invalidates objective measures of time. But does it matter?


4 – Method

Your requirements drive your method

Onsite usability

test

In-person usability

test

Remote moderated

test

Remote unmoderated

testSurvey

Or something else entirely…

Tip: The answer almost certainly is *not* a focus group


5 – Participants

Who will you get to participate in your test?


5 – Participants – How many?


5 – Participants – How many?

5 is the classic answer, and it may or may not be right for your situation.• Practical experience is that you do start to see a lot of

repetition about here• Repeating “like” tasks gives you a lot more data • An explicit assumption is that you will iterate – this is not

a 1-time activity never to be repeated


5 – Participants – Representativeness

“Just go out to a coffee shop, buy someone a coffee, and sit down with them for 5 minutes”

– Almost everyone on every blog on the internet



Do people you will encounter in a coffee shop represent the important characteristics of your users?• If so, great! Head down to the coffee shop of your choice• If they are owner-managers of scrap metal merchants,

consider whether they are likely to be hanging out in Starbucks



One of the single most important things you can do to generate valid insight is recruit people who represent your target users• What are the important characteristics of your users (for

this test)?• Where can you find people like that?• When will people like that be available?• Will you need to incentivise them?• Are there any special ethics? (Children, etc)


6 – Practicalities

What do you need to sort out to make it happen? (1)• A space somewhere• A time• Equipment and software• Script• Note recording sheets and/or recording software• Task sheets for people to follow• Participant availability (and overbooking)• Prototypes, live sites, login details…


6 – Practicalities

What do you need to sort out to make it happen? (2)• Attendees to help you

• Remember you are trying to stimulate valuable action. Who can help make action come about? (A: People who set priorities and choose what work to do)

• Development Managers• Product Managers• Developers• Etc…


Example

1. A plan

2. A “method” (prototype)


Doing


This morning, 9am

– Remote, moderated test

– Web-hosted prototype plus alpha build

– Skype with video, + Evaer to record

– Note recording sheets (formatted)

– Participant sent task sheets and URLs in advance

– Chris – pictured – co-moderating & note-taking


Running a test - Prepare

Be ready

Do run-throughs in advance to check everything works

Make sure all co-moderators know what’s going to happen and what they need to do

Think about if you need to do “resets” between participants if data or state can be persisted

Allow yourself time to set things up


Running a test - Introduce

Usability tests can be stressful for the participant• It’s not them on test – it’s the product or site• Everything that goes “wrong” is the fault of the designers,

and the most useful bits of the session• People just like them are particularly interesting – you

want to know how easy they find it to use• Reprise what to expect, and how long the session will take• Get any permissions (e.g. informed consent, consent for

recording etc…)


Running a test - Prepare

Recommendations• 1 moderator (talker), 1 note taker

• Lets one person remain engaged and keep the conversation going

• Should result in better notes• Notes vs. audio/video

• Need to be clear on what makes good notes• Notes are faster to analyse than video, if they’re good• Video is good to go back to if the notes are lacking


Running a test – Run!

Talk aloud?• Probably – good insight• Completely invalidates any objective measures of

efficiency (but does it matter?)• Might not be appropriate for tasks where “flow” is

important


Running a test – Dealing with questions


Running a test – Questions

DO NOT ANSWER QUESTIONS


Running a test – Questions

Some suggestion for avoidance• “What do you expect that to do?”• “What do you think you could do next?”• “If I wasn’t here, what would you do?”• “Try doing whatever you think might help”

• If necessary, admit you’re being awkward and unhelpful, but you need to know what they’d do if you weren’t here

• Let people struggle for a while, but rescue them before they’re suicidal


Running a test – Finish

Show your appreciation• You should be grateful• Be grateful• Be genuine


Analysis and reporting


Analysis and reporting

For another day…but remember

We are doing this to stimulate valuable action.

NOT:


Thanks

Chris Collingridge@ccollingridge

planning and running usability tests

Design

stimulate

interaction

people enter

level interaction

coffee shop

important

4 methodsome

formative