© simeon keates 2008 usability with project lecture 11 – 21/10/09 susanne frennert

© Simeon Keates 2008

Usability with ProjectLecture 11 – 21/10/09Susanne Frennert


User trials – how to prepare for them


The importance of user trials

User trials are the “gold standard” for finding usability issues

They can be expensive to set-up

You need to plan to ensure maximum benefit from them


Prepare, prepare, prepare

As the old mantra goes:

Fail to prepare, prepare to fail

It is essential to plan well

You need to identify:• What do we want the users to do?• What with?• What results are we expecting to collect?• How are we going to analyse them?• What are we going to do with them?• How can things go wrong???


Preparing for a user trial…

The next slides show an corporate-centric view of how to do usability Source: “Observing the user experience” by Mike Kuniavsky

Issues to consider:• The steps described are very typical in industry• However, are they the “best”?


Preparing for a user trial…

Stage 1: Deciding what to do

Stage 2: Setting up a schedule

Stage 3: Estimating budgets

Stage 4: Preparing the research plan

Stage 5: Maintaining the research plan


Stage 1 – Deciding what to do

Step 1 – Collect issues and present them as goals

Step 2 – Prioritise the goals

Step 3 – Re-write the goals as questions to be answered


Step 1 – Collecting issues and presenting them as goals

1 – Identify the stakeholders, e.g.• Product development managers• Interaction designers / information architects• Marketing• Customer relationship managers• Corporate branding• Users!!!


Collecting issues and presenting them as goals

Examples of key questions to ask:• 1 – In terms of what you do on a day-to-day basis, what are the goals of the

product?• 2 – Are there ways your current solution is not meeting those goals? If so,

what are they?• 3 – Are there questions you want answering about the new product? If so,

what are they?

Make a list of the accumulated goals…


Step 2 – Prioritising the goals

You should now have corporate and user goals

The prioritisation of those goals may be “obvious”

If not, you need a method to help you decide


Prioritising the goals

List the goals Rate each for “importance” (1 = unimportant, 5 = must have) Rate each for “severity” (1 = annoyance, 5 = showstopper) Multiply together and rank by resultant score

Example:

Goal Importance Severity Priority

Users cannot currently complete purchases on our site

5 5 25

Users cannot find the HELP functionality 3 3 9

Better conversion of viewers to shoppers 5 3 15


Step 3 – Re-write the goals as research questions

Issue:• Better conversion of viewers to shoppers

Research Question:• Why don’t [some] visitors become shoppers?

Issue:• Help people use the search engine “better” and more often

Research Question:• How do people navigate the site, especially when looking for something specific?

Issue:• Why are so many people abandoning their shopping carts?

Research Question:• How do they expect the shopping cart to function? Is it failing them?


Expand general questions with specific ones

Example: Why are so many people abandoning the shopping cart?

Specific questions:• What is the ratio of carts abandoned to those completed?• On what pages are they abandoned?• What pages do people most frequently open their shopping carts?• Do people understand the instructions on the cart pages?• Do they know they are abandoning the cart?• Do they know what a shopping cart is?• How do they use the cart?• How do they shop on the site?


Tips for setting the research goals

Never go into user research to prove a point

Never create goals that seek to justify a position or reinforce a perspective• The aim is to uncover what people really want

Learn the product thoroughly• Research goals are more meaningful is you understand what is truly needed

Be prepared to address fundamental questions about the product• Even if the question is: “Should we be building this product?”


Stage 2 – Setting up a schedule

You need to turn the goals into a research plan

Need to add a time-line (schedule) to the goals

Needs to be sensitive to the development schedule


Option 1 – In the beginning… (of a development cycle)

Early design and requirements gathering• Internal discovery: identify the business requirements and constraints• Survey: determine (e.g.) the demographic segmentation and product use of

the existing user base• Log file analysis: examine the users’ current behaviours (if such data

available)• Profiling: develop a representation of the users based on existing users or

those of competitive products• Usability testing: uncover current interaction problems with existing product• Contextual enquiry: uncover problems users have with the product and the

task• Task analysis: specify how he problems are currently solved (or not)• Focus groups: determine whether people feel the proposed solutions will help


Option 1 – In the beginning…

Development and design• Usability tests: perform 4 to 5 back-to-back usability tests of prototypes to

test their efficacy• Competitive usability tests: compare the prototypes to competitors’ products

to determine strengths and weaknesses

Post-release• Surveys and log file analysis: compare change in use of the product

compared with past behaviour• Diaries: track long-term behaviour• Contextual enquiries: study how people are actually using the product


Option 2 – In the middle…

Usability is not always recognised as an issue up-front

Decisions will have been made about:• Who the users are• What their problems are• What solutions to use

These often cannot be revised without significant costs

This is often about “making the best of a bad job”

If usability is raised as an issue midway – then that usually means a problem has been found


Option 2 – In the middle…

Design and development• Usability testing and competitive usability testing: rapidly iterate and improve

the design

Post-release• Log file analysis: perform analysis before and after release to see how user

behaviour has changed

Requirements gathering• Contextual inquiry: identify outstanding issues from existing user base for the

next product release


Organise research questions into projects

To establish a schedule, it is necessary to understand the work packages to be performed

Need to associate time estimates with research questions• e.g. What is the ratio of carts abandoned to those completed?• Time to answer: 2 days of log analysis

Need to aggregate (collect) those time estimates into work packages

Then order those work packages into projects

Then produce a timetable (schedule) for the work


Choosing between the techniques

Technique: Profiles (c.f. Personas)

Stage of development: Beginning Duration: 2 – 5 days over 2 weeks Cycle time: Once per major design, or when new user markets are

defined

Description: Turn audience descriptions into fictional characters to understand how user needs relate

Benefits: Low cost method that creates good communication tools. Brings focus onto specific audiences rather than “the user”

Pitfalls: Based primarily on team’s understanding of users, not external research



Technique: Contextual inquiry and task analysis

Stage of development: Initial problem definition Duration: 2 – 4 weeks (not including recruiting) Cycle time: Once per major set of features

Description: Observe people as they solve problems to create a mental model that defines their current understanding and behaviour

Benefits: Creates a comprehensive understanding of the problem that is being addressed

Pitfalls: Labour intensive



Technique: Focus groups

Stage of development: Early development feature definition Duration: 2 – 4 weeks (not including recruiting) Cycle time: Once per major set specification, then after every feature

cluster

Description: Structured group interviews(?) of 6-12 target audience representatives

Benefits: Uncovers people’s priorities and desires, collects anecdotes and investigates reactions to ideas

Pitfalls: Subject to “group-thinking”, desires can be easily misinterpreted as needs



Technique: Usability testing (c.f. usability observations)

Stage of development: Throughout design and development Duration: 1 – 2 weeks (not including recruiting) Cycle time: Frequently (ideally)

Description: Structured one-on-one interviews(?) with users as they try specific tasks with products/prototypes

Benefits: Low-cost(?) technique that uncovers interaction problems Pitfalls: Does not address underlying needs(?), just abilities to perform

actions



Technique: Surveys

Stage of development: Beginning of development, after launch and before re-design

Duration: 2 – 6 weeks Cycle time: Once before major (re-)design, regularly thereafter

Description: Randomly selected representatives of the audience are asked to complete questionnaires, quantitative summaries are then generated

Benefits: Quantitatively describes the audience, segments them, investigates their perceptions and priorities

Pitfalls: Does not address why people have those perceptions or what their actual needs are, subject to selection bias



Technique: On-going research

Stage of development: Throughout life of product Duration: On-going Cycle time: Regularly after release

Description: Long-term studies of users, done through diaries and advisory boards

Benefits: Investigates how users’ view and use patterns change with time and experience

Pitfalls: Labour intensive, requires long-term participation



Technique: Usage logs and customer support

Stage of development: Beginning of development, after launch and before re-design

Duration: Varies Cycle time: Regularly after release

Description: Quantitatively analyse Web server log files and customer support comments/throughput

Benefits: Does not require additional data gathering, data reveals actual behaviour and perceived problems

Pitfalls: Does not provide any information about reasons (why) for behaviour or problems


Stage 3 – Estimating budgets

Budgets are not just about finance You need to consider:• People’s time (including yours and your team’s)• Recruiting and incentive costs• Equipment costs• Travel costs• Lab costs• etc.


Example budgetary estimates

Task Time

Preparation for a single research project

10 hours

Recruiting and scheduling 2 – 3 hours per person recruited

Conducting research

• Contextual Inquiry / task analysis

• Focus groups

• Usability tests

5 hours per person

3 hours per group

3 hours per participant


Example budgetary estimates

Task Time

Analysing results

• Contextual Inquiry / task analysis

• Focus groups

• Usability tests

5 hours per person

4 hours per group

2 hours per participant

Preparing a report for e-mail delivery 12 hours

Preparing a 1 hour presentation based on report

6 hours


Stage 4 – Preparing the research plan

Summary• Describe the research plan in 2 paragraphs (<200 words)

Research Issues• What are were the identified issues that the research needs to address• Example: “Based on conversations with [x, y and z] we have identified 5 major

issues that this research will attempt to shed light on…” etc.• Then describe the high-level issues (not specifics)

Research Structure• Describe the actual research methods that you will employ and the issues that

they will address• Example: “We will conduct four sets of one-on-one structured interviews to

establish [x]. We will then perform…“ etc.


Preparing the research plan (continued)

Schedule• Include a detailed schedule, ideally laid out on a week-by-week basis• Include prominent milestones

Budget• Include the projected budget• This should be broken down into work projects• And further broken down by major activity• Example: “5 usability tests: Preparation 10 hours; Recruiting and scheduling 40

hours (assuming 40 participants); Conducting tests 120 hours; Analysing tests 80 hours… Total time: [x] hours”

• Also include financial costs where known• Example: “Recruiting incentive (25-40 people) $2500-$4000; Supplies (food,

videotape, etc.) $500… Total cost: $[x]”


Preparing the research plan (continued)

Deliverables• Specify what you are going to produce at the end of the research• Example: “The results of each usability test will be sent via e-mail as they

are completed. Each e-mail will include an outline of the procedures, a profile of the people involved, a summary of all trends observed in their behaviour (as they relate to the initial research goals), problems they encountered and a series of supporting quotations.”


Stage 5 – Maintaining the research plan

The research plan is a “living” document

It needs to be re-visited, reviewed and revised to keep up-to-date

Especially as preliminary information and results begin to come in from the users


Improvements to this approach

The preceding slides were based on Kuniavsky’s “Observing the user experience”

These are not “definitive” but “indicative”

Remember, for the best research plan: “It depends!”


Additional steps to consider

Pilot studies• This discussion has no reference to conducting pilots

What if something goes wrong?• Related to pilot study issues• What if a show-stopping error is uncovered early on?• What if the functionality is not going to be completed on time? (e.g. coders

still coding)

What exactly are we testing and how exactly are we going to report it?• The method described here is (arguably) too imprecise and unspecific• Method is based on answering perceived issues• Is that good enough?

Think of this as the “marketing” version of the research plan


Yet further points to consider

Where is the testing to be conducted?• Your office, remote, 3rd party venue?

What forms of data collection are to be used?• Logging, video recording, note-taking, etc.

What approval is required?• Ethical approval, medical approval, etc.


A more precise approach…

A standard “scientific” report/paper has the following structure: Aim: What are we trying to do here (one paragraph max) Hypotheses: What exactly are we examining? Background/rationale: What is the context of this research? Equipment: What are we using? Method: How are we going to do this? Results: What data did we collect? Discussion: What does our analysis show? Conclusions/summary: What did we learn? Further work: What should we do now?

Note: this is the basic structure for your final report!


Even more precise

A good plan should specifically include:

The overarching issues (Aims)

The exact questions being asked (Hypotheses)

The expected data types to be collected to test those hypotheses

The mathematical analyses to be used on those data types

A pre-determined set of success/failure criteria


An example

The overarching issues (Aims)• Too many users abandon their shopping cart / do not complete their purchases

The exact questions being asked (Hypotheses)• Hypothesis 1 (H1): Our site has a significantly higher rate of abandonment

than AN Other site • [Note – need to define “abandoned”]• Hypothesis 2 (H2): The rate of abandonment is a direct consequence of

usability/accessibility issues

The expected data types to be collected to test those hypotheses• H1: (From log files) Number of purchases completed vs. number abandoned• H2: (From usability trials) Quantitative and qualitative user data


An example (continued)

The mathematical analyses to be used on those data types• H1: Count number of purchases completed and number of purchases

abandoned; Identify which purchases abandoned now are completed at a later date; Compare with available data from other sources (competitors, older versions of site, performance goals); Test for statistical significance at 5% threshold

• H2: Ask users to complete a number of different purchase types; Count number/proportion of trials completed successfully; Count number of errors (and severity) encountered; Record time for task completion; Record user satisfaction with task (on Likert-type scales); Report severe/frequent usability and accessibility issues and user dissatisfaction

A pre-determined set of success/failure criteria• H1: Must exceed competitor X or defined performance goal [X% conversion]• H2: No severe usability/accessibility issues; Users generally satisfied


Summary

Ideally, need a blend of the “scientific” plan with the “business” plan for a working document

However, may need a “sanitised” version for non-experts


Involving users in the design process

“Know the users - for they are not you”


Users…

Who are they?• Varies by project

What capabilities?• Varies by project

How many?• Varies by project

How to report the results?• Varies by project


Question: How many users do you need to test with for a usability test?


Why don't we do large numbers in usability testing?

We are looking for behavioral based insight (what they do).

Statistics tell half the story and often are devoid of context (e.g. Why did they fail?)- Also one of the major problems with gaining insight from web analytics (website traffic statistics).

Our objective is to apply findings to fix design problems in a corporate setting (not academic analysis).

Research shows that even with low numbers, you can gain valid data.


Recruiting users

It is essential to get the right people in

Otherwise your research will be meaningless and possibly dangerous

The next few slides outline the mechanics of how to recruit users


(Idealised) Timetable for recruitment

Timing Activity

t-3 weeks to t-2 weeks Determine target audience

t-2 weeks Recruit initial pool or pre-screen initial group from a database of qualified participants (if available)

t-2 weeks to t-1 week Screen for final candidates

t-2 weeks to t-1 week Send invitations to primary qualified candidates

t-1 week Send invitations to secondary qualified candidates

t-3 days Create schedule for all candidates and contact list for alternatives

t+1 day Follow-up with participants and researchers


Picking your audience

You should already have an audience profile before beginning recruitment

However, you will need to refine it at this stage You will typically need the following:• A demographic profile (age, physical characteristics, education)• A capability profile (functional capabilities)• A product use profile (e.g. Web use)• A technological profile (use of “technology” in general)• Other aspects (e.g. behaviour)


An example profile

Demographics:• 25-35; Male or female; College educated; Income: $100k +

Web use:• Must have a personal computer at home or work• Must use the Internet• Must have 1 or more years of Internet experience• Must use the Internet at least 3 hours per week for personal tasks• Should have some experience with Web productivity applications (e.g.

calendars)

Behaviour• Should regularly hold social events with at least 3 other people• Social events should involve explicitly inviting people


Finding your audience

Start with your personal database• People you’ve worked with before or have recruited

Try your friends and family• Includes your work colleagues

This will see you through standalone projects


Finding your audience

For larger or more frequent projects, you will need to expand your horizons:• Community e-mail lists and on-bulletin boards, but do not spam!• Neighbours (from other companies nearby)• Your users (have an interest in improving the product)• New employees (not as experienced, but motivated)• Past research participants (known to be reliable and interested)• Adverts (post on-line and specify location, incentives, etc.)• Traditional methods (classified ads in local papers)• Day-care centres, charities, etc. (great for accessibility testing)


Additional recruiting tips

Rather than send a very long e-mail, point users to a web-site• Can also use the site for them to register their interest

Clearly state when and where• You usually want local people, unless doing a remote test

Keep track of when and how people found you• So you can streamline further recruitment efforts

Update all contact details every 6 to 8 months


Additional recruiting tips

Try to keep the number of repeat recruits down• Overuse is not good

If someone gives good feedback, add that to your database• These users are great for “short notice” jobs

If time allows, perform a pilot study• Make sure the users and tasks are a good match


Registering an interest/pre-screening

Once someone expresses an interest, you need to follow that up

Prepare an e-mail that:• Thanks them for their interest in improving the product• Summarises how exciting this project is• Asks initial pre-screening questions

Based on that information, classify candidates as either:• Primary (really should involve)• Secondary (good to involve, e.g. may be similar to others in primary list)


Screening candidates

Aim is to find candidates that genuinely match the research interests• Not just match the demographics, etc.

Screening is typically done over the phone or by e-mail• Phone is best, but more time-consuming (person may not answer, etc.)


Screening tips

Stick to 20 questions maximum• Possibly fewer depending on the quality of your pre-screening

Make it short• Only 5 to 10 minutes

Be clear and specific• Make sure the person fully understands what is being discussed

Never use jargon• You may frighten people away, plus see above point

Build in flexibility• Avoid eliminating people on technicalities


Screening tips (continued)

Every question should have a purpose• Time is money, use it wisely

Order your questions from general to specific• Ask the “big” elimination questions first and then work down to more exacting

requirements

Questions should not lead• You should know to avoid such questions already

Clearly state the format of the research• What is to be done and why

Terminate the screening if candidates breaches any of the requirements• Examples: too old, wrong education level, does not use the technology, etc.


Scheduling

The tighter your deadline, the harder the scheduling

Establish scheduling windows• Based on availability of equipment, testing venue, required observers, etc.

Schedule your primary candidates first

Fill in the blanks with the secondary candidates

Try to accommodate your participants’ requirements as much as possible• It’s good manners• And encourages them to come back again


How to schedule users

Write the invitation (when/where, etc.) Send to primary then secondary candidates Receive responses Confirm primary then secondary candidates Send thank-you notes to those not scheduled Make confirmations calls by phone and/or e-mail to all participants the

day before their scheduled appointment Create and distribute a list of participants and times to the research

team Make and put up signs to the testing facility


Including users for accessibility testing

Functional impairments affect each of the stages of preparing for and running usability trials

We will examine how in more detail on Friday…


Exercise


Exercise – part 1

Prepare your research plan for investigating the usability and accessibility of your original and revised web site designs

Your plan must clearly include:• Summary• Research Issues (Aim / hypotheses to be tested)• Research Structure (Method)• Schedule (Method)• [ Budget (Method) ]• Deliverables (Results)

Also, prepare a 10 minute presentation to the “board” for Friday


Exercise – part 2 – Kuniavsky’s stages

Stage 1 – Deciding what to do• Collect issues and present them as goals (Who are the stakeholders? What

are there issues? Etc.• Prioritise the issues (importance * severity)• Re-write the goals as questions to be answered

Stage 2 – Setting up a schedule• You will have a minimum of 4 afternoon slots and 2 morning slots available

for your user trials + data collection + analysis + writing final report - use your time wisely!

• You will be expected to test both versions of your web-sites with at least 4 users - one of whom should be blind (simulated or not)


Exercise – part 3 – Kuniavsky’s stages

Stage 3 – Estimating budgets• Not really that relevant here• How many cups of coffee?

Stage 4 – Preparing the research plan• More on this on the next slide

Stage 5 – Maintaining the plan• You will have opportunity to revisit and revise your plan over the next few

weeks


Suggestions Summary Research Issues (Aim / hypotheses to be tested)• Does the new web-site offer increased usability and accessibility over the

original design?• Add more aims!

Research Structure (Method)• Heuristic evaluation• Accessibility testing plan• Usability trials, etc.• Add more detail here!

Schedule (Method)• First two parts already completed• Usability schedule – add more detail here!


Suggestions (continued)

Budget (Method)• Not really applicable

Deliverables (Results)• Add more detail here!• What do you think you will test?• What quantitative tests will you use?• What qualitative tests will you use?• What measures will you use?• What do you think your success/failure criteria should be?

Remember – the more specific and explicit you make the plan, the easier the usability testing and report writing will become!!!

© simeon keates 2008 usability with project lecture 11 – 21/10/09 susanne frennert

Documents