user research & usability testing, the key to user-centered web site design

Architects of the User Experience

Presented to Potomac Forum Symposium

Building Better Government Web Sites: From Strategy to Fulfillment

March 30, 2010

User Research and Usability Testing:

Techniques and Tips for Optimizing Government Web Sites

Toral Contractor, Director of User Research

Kirsten Miller, Information Architect

2NavigationArts, LLC

Potomac Forum | March 30, 2010

User Experience Design

User Research

• Methods

• Case Studies & Tips

Usability Testing

• Methods

• Case Studies & Tips

Project Lifecycle

Wrap-Up

Agenda



User-centered design, which puts users rather than organizational or system-related

considerations at the center of the design process, is the foundation of our UX

practice.

Information Architecture

• Structure information and functionality for easy access

• Design effective navigational systems and taxonomies

Interaction Design

• Design complex online operations so they are intuitive for users

• Establish clear priorities for user action

Creative Interface Design

• Create an engaging, attractive and brand-building visual interface

• Create a visual hierarchy to facilitate browsing and scannability

User Experience (UX) Design




UX Design Inputs

User Research

• Direct vs. indirect

Usability Expertise

• Expert heuristic assessment

Usability Testing

• Live site vs. prototype

• Moderated vs. unmoderated

• In-person vs. remote

Business Goals

• Stakeholder interviews

Our focus today:




User Research



Direct Methods

Contextual inquiry

• Interview and observe users in their own environment as

they use the site and perform relevant tasks

Intercept interviews

• “Hi. Would you like to help us improve your web site by

answering a few questions?”

Focus groups

• Structured group interviews used to collect opinions and

insights into user needs and perceptions

User surveys

• Collect a range of qualitative and quantitative data on user

demographics, behavior and opinions

Card sorts

• Users sort, group and label site topics

User Research

What do

YOU think?

In a nutshell:



Case Study: Contextual Inquiry

Department of State – Bureau of Consular Affairs

Intranet redesign

• Shadowed employees at work

• Inventoried common user tasks both on and off web site

• Asked about browser bookmarks and paper documents

kept close at hand

• Observed ways of browsing and using information

• Inventoried likes and dislikes about current site, wishlist

for redesign

Benefits:

• Understand the environment they work in. Are they

under a lot of stress while working? Do they have a

short amount of time to search for something?

• Opportunity to observe documents they have printed out

at their desks for easy access and reference for

common tasks.

UX DESIGNDEVELOPMENT

INTEGRATIONDEPLOYMENT SUPPORT

User Research: Direct Methods



Contextual Inquiry Tip:

Pay Attention to the Sticky Notes!



M.D. Anderson Cancer Center

• Interviewed patients and family members in

the Learning Center who were seeking

information or passing through on their way to

New Patient Orientation.

• Discovered two modes: Research mode (e.g.,

seeking information about a disease, the

organization, or a doctor) and Task mode (e.g.,

quickly print a map or schedule).

George Washington University

• Asked students about their primary reasons for

applying, attending or leaving GW

• Gained understanding of current student

perspectives and expectations of GW

Benefits

• Quick and economical way to sample a target

population

Case Studies: Intercept Interviews






Case Study: Focus Group




Nemours

• Interviewed parents of child patients

• Learned of two main types of needs with patient

families: Long-term (when a child has cancer),

and Short-term (when a child has a minor

condition).

• Learned that families weren’t expecting the

hospital to provide information about diseases,

but did want hospital-recommended or approved

sources of information.

Benefits

• Able to interview a large group at once for their

opinions, evaluation or formulation of a concept.

• Group discussion triggers conversations and

topics that may not occur in one-on-one

interviews.

• Pick up on common user language, terminology



Focus Group Tip:

Get written feedback from participants before starting group discussion.



Case Study: User Survey


• ForeSee Survey implemented

• Questions about user roles, goals, satisfaction,

problems encountered, and suggestions for

improvement

• ACSI model questions allow customer satisfaction

benchmarking against other ForeSee clients

Benefits

• Large sample of quantitative data and qualitative

feedback to guide and support design decisions

• Customer segmentation helps refine and prioritize user

profiles




User Survey Tips:

• Form questions around user tasks

• Use survey logic to ask targeted

questions to various audiences

• Include some open-ended

questions



Case Study: Card Sort




The George Washington University

• Open sort: High school students and

prospective undergraduate students grouped

and labeled undergraduate majors

• Closed sort: Freshman evaluated main

navigation of the website

Nemours

• Open sort: Groups of patient families

categorized Nemours content to build the main

navigation.

• Learned that all the participants wanted a “My

Account” area for easy access to payment

options, history of patient activity, appointment

information, and an easy way to communicate

with the doctor.

Benefits

• Formulate and evaluate navigation that aligns to

users’ mental model of the information



Card Sort Tips:

• Conduct open card sort first

• Then conduct closed sort to

validate design



Indirect Methods

Web Analytics

• Analyze site traffic and user behavior

Call center interviews and log analysis

• Interview call center agents and review call

logs, if available

Social media sites

• Explore groups that form around common user

interests; review user behavior on your own

social media sites/pages

• Can also use for direct user research

User Research

In a nutshell:

What are

THEY doing?



Case Study: Web Analytics


intranet redesign

• Number of visitors

• Most popular pages and directories

• Most popular entry and exit pages

• Top referrers/inlinks

• Most popular search terms

• Time on site

• Pages per visit

Benefits

• Understand user behaviors and priorities

• Set success metrics/benchmarks to evaluate post-

launch

• Identify areas for future qualitative research



User Research: Indirect Methods



Web Analytics Tip:

Look at exit pages for areas to focus on for

further qualitative research



Case Study: Call Center Analytics



User Research: Indirect Methods

NHTSA

• Interviewed call center personnel about

common questions and emails from the public

• Gained insight into customer needs not

currently being met by the web site

Benefits

• Call center agents are the front line for direct

interaction with the public

• Agents often walk customers through the web

site and know where the pain points are



Call Center Analytics Tip:

Ask agents about browser

bookmarks and FAQ lists.



Case Studies: Social Media Sites



User Research: Direct & Indirect Methods

Indirect: Council of Residential Specialists

• Networking is very important to realtors and

therefore, social networking sites was a key source

for information

• Researched social networking sites to see what

users are discussing, gain insight into their needs

and interests

Direct: Smithsonian

• Help imagine a completely new and more engaging

way for people from around the world to interact with

the Smithsonian Institute online

• Used YouTube to solicit feedback in forms of videos

to understand what users wanted to see from

Smithsonian



Social Media Tip: Sign up for Google alerts so

you know what people are

saying online about your

organization.



Usability Testing



Usability Testing Techniques

Live site vs. Prototype testing

• Live site: Test participants interact with currently existing site

• Prototype: Test participants interact with proposed design; can be

low or high fidelity, paper or electronic

Moderated vs. Unmoderated

• Moderated: Test participants interact with system/prototype and

with researcher/facilitator

• Unmoderated: Test participants interact only with

system/prototype; researcher may observe behavior and/or

analyze data collected by system

Remote vs. In-person

• Remote: Test participant and researcher/facilitator in separate

physical locations

• In-person: Test participant and researcher/facilitator in same

physical location

Usability Testing

Mix and match!



Case Study: Unmoderated, Remote, Prototype Test

Department of State – travel.state.gov

• Testing navigation menu prototypes using an

online heatmap generator

Benefits

• Participants complete test in their own

environment and whenever they have time

• No need for travel or appointments

• Inexpensive

• Many tools available now for creating remote,

unmoderated usability tests and analyzing

data generated

PLANNING &

DISCOVERY

DEVELOPMENT


Usability Testing



Case Studies: Moderated, Live Site Test



Usability Testing

Remote: World Bank

• Tested search feature on live public site with

domestic and international users within the

international development community and media

• Conducted free exploration test in which users

formulated their own scenarios true to the way they

use the site

In-person: Department of State - travel.state.gov

• Set specific tasks with correct and incorrect

outcomes; scored and analyzed results

Benefits

• Directly observe user behavior on the site; identify

trouble spots

• Draw out user opinions and insights about

performing tasks while they are doing it



Moderated Live Site Test Tip:

International users?

Plan to work odd hours!



Case Study: Moderated, Remote, Prototype Test

KidsHealth.org

• Tested a low-fidelity, clickable prototype with

children, teens, parents and teachers to

evaluate design at wireframe stage

Benefits

• Users test in their environment or the context

in which they would use the site

• No need to travel

• Lower participant incentives

• Many tools on the market now to facilitate

remote user testing

PLANNING &

DISCOVERY

DEVELOPMENT


Usability Testing



Moderated

Remote Test Tip:

Be prepared for

distractions



Project Lifecycle



User Research

Contextual inquiry


Focus groups

User surveys

Card sorts

Web analytics

Call center interviews, log analysis

Social media sites

Usability Testing

Live site testing

Prototype testing

When?

Project Lifecycle

Architects of the User Experience

Presented to Potomac Forum Symposium

Building Better Government Web Sites: From Strategy to Fulfillment

March 30, 2010

User Research and Usability Testing:

Questions?

Toral Contractor, Director of User Research

Kirsten Miller, Information Architect

user research & usability testing, the key to user-centered web site design

Design

user research navigationarts

user researchpotomac

user needs

user roles

direct methods potomac

common user language

potomac forum symposium

perceptions user surveys