Usability Assessment, Round Two:Re-Examining the UW Libraries

Information Gateway

Jennifer WardUniversity of Washingtonjlward@lib.washington.edu

Presentation Overview

Background and History Current environment Hardware, personnel Efforts to date Lessons learned and futures

Looking Back Move to Web interface for most

services – early 1998 Rapid design, testing, and prototyping

of website Only formal usability testing was by

students as a class assignment

Current Environment

Web Steering Committee (WSC) established August, 2000

Decentralized website management Approximately 170 staff in 40 units with

write permissions No testing on current interface

Lab and Support Usability lab

– Other labs on campus– Currently in a shared conference room; will

have dedicated space soon– PC with Camtasia, video camera

Half-time graduate student dedicated to usability efforts– Brought a fresh perspective to the process

Work with WSC to determine priorities System-wide support for assessment

Summer, 2001 Online Survey

July 23 – August 13, 2001 49 unique responses; almost 1/3 each

faculty, students, staff Goal was to get a sense of what users

thought of the Gateway Based on a previous survey Used WebQ survey software, linked

from home page

Summer, 2001 Survey Results Asked a lot of open-ended questions

about navigation and features Nearly 80% successfully found what

they wanted– “Once one becomes familiar with the

site, it’s easy to navigate.” Library Catalog most important resource

– Most requested new features focused on catalog functionality

Points to Ponder

Timing is everything!– Low response rate

Data quality– Many open-ended questions went

unanswered– Too focused on navigation and features;

need to broaden it

Autumn, 2001 Online Survey September 21 – October 20, 2001 Same goal as before, but do a better job

this time Plugged in online campus newsletter Incentive was provided, but not

advertised 131 unique responses

– 10% faculty, 28% staff, 27% grad students, 27% undergrads, 5% alumni, and 2% non-affiliates

Autumn, 2001 Survey Structure Restructured summer survey so we’d

get more answers Ran past a couple of local experts for

feedback 15 questions, multiple choice or open-

ended Users could remain anonymous

– Didn’t get affiliation, which would have been nice for data analysis

Autumn, 2001 Survey Results

Participants for follow up studies Part of site that drew the most

comments was our catalog (!!) Overwhelming response – it’s not

“broken”– Almost 80% very satisfied or satisfied– Some areas need work and we’re

addressing them– “Micro” level changes

Autumn, 2001 Survey Results

Asked what new feature users wanted:– Some things were already on our site;

looked at making them more prominent– Database selection wizard/helper– Online tutorial for Gateway and catalog– Better integration with campus

authentication– $100 put into their bank account

Autumn, 2001 Survey Results Data clumping Wording and Navigation Databases and Journals

– Show vendors on database list– Need an “all-purpose” database instead of

having so many– No clear winner on hierarchical vs

alphabetical listing of databases and e-journals

Card Sorting November 13 – December 6, 2001

– Developed and started recruiting while we were waiting for online survey to conclude

To determine the significance of features of the Gateway and how users would organize and name these features

Standardized procedures and documentation– Consent forms, scripts, task lists

Participants– one undergrad, one grad, one faculty, and

two staff members (not what we wanted)

Card Sorting – Tasks Pilot Test Think aloud protocol, videotape, index

cards Two sets of tasks with 46 cards Rank according to perceived

importance– Students more flexible than faculty/staff

Group according to their own navigational structure– Change wording if they wanted

Card Sorting – Results Statistical analysis (dendrogram)

– More significant with more participants Items in upper right are more important Confusing wording (i.e., Cascade,

Connecting, Starting Points) Didn’t understand News and Current

Events Have not used data in redesign

“By Subject” Pages One of the “broken” items from the survey

and WSC’s perspective Politically hot topic How to help beginning researcher get started,

also try to address the layout Hierarchical vs alphabetical presentation Recruitment

– Targeted email, undergrads from library units

http://www.lib.washington.edu/subjects/

“By Subject” Pages – Tasks Think aloud protocol, captured screen

activity with software Write answers to questions as they go

along Mockups on a test server Undergrad tasks varied – did they use

“Core Resources”? Graduate student and faculty tasks were

targeted to their discipline

“By Subject” Pages – Results Undergraduates

1. Wanted listing by class name, searched Yahoo! for everything (more comfortable)– Wanted ranking system (4 stars)

2. Expert researcher who never left the test pages; completed tasks in 2-3 minutes– Understood and liked “Core Resources”

3. Stayed with test pages except when he felt lost, then returned to his favorite search tool – Google– Drawn to resources with the folder icon

“By Subject” Pages – Results

Graduate Students1.Went there once, wasn’t satisfied and

never came back2.Not inclined to leave the pages; weakest

researcher Faculty

1.Resources were already bookmarked

“By Subject” – Other Efforts

We’re working on adding a survey to the bottom of these pages

Search box on each page Simplify the design by removing

explanatory blurbs at top

Lessons Learned

However long you think something will take, multiply the time by 3 and you might be close

Standardize the process as much as possible; use templates and set guidelines

Refine tests before the involving users – run pilot tests or talk with experts

Timing, timing, timing Try not to put something online unless you

know it works. If users don’t find what they’re looking for the first time, they don’t come back.

Questions?

The website:http://www.lib.washington.edu/usability/

Contact us:jlward@lib.washington.edu or libuse@u.washington.edu