preparing and running user experiments

Preparing and Running User Experiments

By Mei Li, Pearl Ho, and Deepika Gandhi

How to Prepare and Run Usability Testing

How to gather participants and prepare a proposal. --Mei

Preparing the environment, test materials, and test team.

--Pearl

How to measure usability and deal with participants.

--Deepika

IntroductionWe cover everything that happens or that you should be aware of prior to collect your data.

Preparing for your User Requirements Activity

Creating a proposal Deciding the duration and timing of your session Recruiting participants Tracking participants Creating a protocol Piloting your activity


A usability activity proposal is a road map for the activity you are about to undertake.

Why create a proposal?

Surprises Assumptions Misconceptions

Tip:Multiple activities, separate proposals

Creating a proposal

History Objectives, measures,

scope of the study Method User profile

Recruitment Incentives Responsibilities Proposed schedule

Creating a proposal

RecruitmentE.g. John Smith from Product team will recruit total 30 participants by an advertisement on the web and a recruitment agency.

Creating a proposal

Incentives

E.g. Participants will receive $75 in AMEX gift checks for the

participation.

Creating a proposal

Incentives

E.g. Participants will receive $75,000 in AMEX gift checks for the

participation.

Creating a proposal

Responsibilities Who is responsible for what task? Key word – Specific

The product team is responsible for recruiting participants.

John Brown from the product team is responsible for recruiting participants.

Creating a proposal

Proposed Schedule Indicate time for each deliverable Key word – Specific People often underestimate the amount of time it

takes to prepare for an activity.Tip:

Request deliverables a week before we absolutely need them. Then, if deliverables are late (which they often are), it's OK.

Creating a proposal

Getting Commitment Email the proposal? Issues:

stakeholders will criticize: Skills/knowledge/objectivity of the person who

conducted the activity – Be a member of the team and earn their respect Participants in the activity Tasks/activity conducted – Getting everyone sign off on the proposal

Creating a proposal

Organize a meeting: Objective of the activity Data which will be collected User profile Responsibility Schedule

Signed

Creating a proposal

Deciding the duration and timing of your session

Group Session5-7pm or 6-8pm bestWith some food, perfectBreak the session into small chunks if

possible

Recruiting participants

Participants numberParticipant incentivesDeveloping a recruiting screenerCreating a recruitment advertisementRecruitment methodsPreventing no-showsRecruiting international participants Recruit

special populations


Participants number Facts to be concerned

Availability Representative

Method – Convenience sampling Use the available sample of the population, instead of

representatives from the population at large. e.g. Research done by college professors often uses college

students for participants. Tip – Identify participants types

Where should you go to Find Participants

For this type of Product Look to this sourceTravel reservation system Travel agencies

Bank teller system Banks

New version of existing operating system

Customer lists

Medical office software Phone lists of local doctors' offices


Participant incentives Mode

Cash, normally not. Store? One of your product for free Gift certificate (an electronics store, movie pass) Gift checks Charitable donations in the participant's name (for highly

paid individuals CEOs) Amount

Make the incentive large enough to thank people for their time and expertise, but nothing more.

Pay everyone in the same session the same amount in case of amount changing.


Developing a recruiting screener Via phone, don't email Work with the product team, instill team-work sense. Keep it short Use test questions, you want honest participants Collect demographic information, once decide a candidate Eliminate competitors Provide important details

Time, date, location, compensation, rules, signature Prepare a response for people who do not match the profile


Creating a recruitment advertisement Provide some details about your study Include date, time, location of the study Indicate key characteristics, not all Don't stress the incentive State how they should respond

e.g. generic email address Be aware of types of bias


Recruitment Methods Advertise on community bulletin board sites Create an in-house database Use a recruiting agency Make use of customer contacts


Preventing no-shows A 10% "no show " rate is common Provide contact information Remind Over-recruit

Confirm the Appointment: Thanks for agreeing to participate The date and time you expect them to be there Where to come, including a map and directions How long to expect to be with you The purpose of the test Reminders about the video cameras Incentive A person's name and a phone number to call if they

have questions or need to reschedule



Recruiting International Participants Agency Cultural and behavioral taboos books Translator Punctuality Holiday concern


Recruiting Special Populations Children, elderly, disabilities

Transportation Escorts Facilities


Recruitment Methods Advertise on community bulletin board sites Create an in-house database Use a recruiting agency Make use of customer contacts

Tracking Participants

Tax implications Avoid professional participant Create a watch list

Tax Dishonesty Poor attendance No show-up

Creating a Protocol

All procedures and their order Act as a checklist for all of the session steps

Piloting your Activity

Check audio-visual equipment working Clarity of instructions and questions Find bug or glitches Attendee

Experienced Product team member

Preparing the environment, test materials, and test team.

--Pearl Ho

preparing and running user experiments

Documents