usability assessment report: american library association website

Usability Assessment Report: American Library Association Website

Submitted to

American Library Association Rob Carlson

50 East Huron Chicago, IL 60611 [email protected]

1-800-545-2433 x3272

Submitted by

Jimmy Vu, Dustin Chambers, Elizabeth Buie, Dick Horst UserWorks, Inc.

1738 Elton Road, Suite 138 Silver Spring, MD 20903

301-431-0500 [email protected]

December 6, 2006

The information enclosed in this document is proprietary and confidential to the American Library Association (ALA). Distribution of this document is limited to ALA and any designates that they may choose.

2

Table of Contents Introduction and Approach ..............................................................................................................1

What Works .....................................................................................................................................2

Major Usability Issues and Recommendations................................................................................2

INFORMATION ARCHITECTURE INCONSISTENT WITH USERS’ MENTAL MODELS .................................... 2 DIFFICULT AND CONFUSING NAVIGATION ............................................................................................... 4 INADEQUATE HANDLING OF MEMBER/PROFESSIONAL/PUBLIC CONTENT .............................................. 6 DIFFICULT AND CONFUSING SEARCH FUNCTION ..................................................................................... 8 POOR VISUAL/PAGE DESIGN .................................................................................................................... 9 INCORRECT OR OUTDATED CONTENT AND LINKS.................................................................................. 11 CRYPTIC TERMINOLOGY......................................................................................................................... 11 LONG, MEANINGLESS URLS .................................................................................................................. 12

Online Survey - Quantitative Analysis ..........................................................................................14

INTRODUCTION ....................................................................................................................................... 14 USER SATISFACTION............................................................................................................................... 14 QUANTITATIVE ANALYSIS...................................................................................................................... 14

Accessibility Compliance ..............................................................................................................16

ACCESSIBILITY EVALUATION METHODS................................................................................................ 16 TEST ENVIRONMENT............................................................................................................................... 16 ACCESSIBILITY SCORECARD................................................................................................................... 17 PRIORITY 1 CHECKPOINTS ...................................................................................................................... 17 PRIORITY 2 CHECKPOINTS ...................................................................................................................... 19 POSITIVE NOTES ..................................................................................................................................... 21 ACCESSIBILITY ISSUES ........................................................................................................................... 21 BROWSER REVIEW.................................................................................................................................. 23

Appendix A – ALA Member Participant Screener........................................................................26

Appendix B – ALA Nonmember Participant Screener..................................................................28

Appendix C – Informed Consent Form .........................................................................................31

Appendix D – Participant Demographics ......................................................................................32

Appendix E – Test Administrator Guide .......................................................................................33

Appendix F – Usability Testing Notes...........................................................................................38

1

Introduction and Approach

UserWorks conducted a usability assessment of the current ALA Web site (ala.org) in September and October, 2006. The assessment comprised three activities:

• Heuristic evaluation • Usability test • Accessibility review

For the heuristic evaluation, we exercised the ALA site ourselves, taking the perspective of members of the audience groups, and noted design aspects that follow accepted practice and aspects that fail to follow such practice.

For the usability test, we brought thirteen representative users (six ALA members and seven nonmembers) into our usability lab to interact with the Web site, using tasks that focus groups had previously identified as most important for each of these audience groups. We observed the extent to which the participants could perform the tasks, recorded the navigational paths that each participant took for each task, and collected their comments. Specifics of the usability test methods appear in Appendix F.

For the accessibility review, we evaluated the site with and without JavaScript enabled, using Firefox and Internet Explorer on Windows XP and Firefox, and using Firefox and Safari on Mac OS X Tiger. We also checked the World Wide Web Consortium (W3C) Web Accessibility Initiative (WAI) compliance by reviewing selected portions of the site with respect to WAI Level II guidelines, using the Bobby automated tool.

In addition, we took account of user comments about ALA site usability that occurred in the context of our recent online survey and focus groups. Although the other findings from these activities will be reported in a separate User and Task Analysis Report, this report covers the usability-related concerns that surfaced during these activities.

This report combines the results from all of the usability data collection methods and does not necessarily distinguish findings of one method from findings of another. It presents the accessibility findings separately from the usability findings, as accessibility has a different scope.

2

What Works

A number of aspects of ala.org are meeting users’ needs. For example, when asked what features the site has that they find particularly useful, survey respondents listed the following:

• accredited schools

• ACRL blog

• activities

• Advocacy (3 mentions)

• ALSC info

• Awards (4 mentions)

• challenged books

• Division info

• downloadable artwork

• eLearning

• Employment

• events

• IF articles (2 mentions)

• left-column links

• links to other sites

• news

• professional development opportunities

• professional documents as resources for graduate students.

• photos of Council members

• promotional materials (2 mentions)

• publications

• registering for conferences

• reviews of new books and research

• ROI info

• store

• toolkits for publicizing annual events

• unique information that I need

• YALSA (3 mentions)

Most of these items, however, seem to arise from the usefulness of the content, more than from the usability of the site or of the content areas.

On a note more directly related to usability, our testing found that participants were generally successful in finding the following information:

• Winners of book awards • ALA’s purpose and the ways in which the organization supports that purpose • Benefits and costs of membership in ALA, and how to join • Upcoming ALA-sponsored programs and events

ALA website users offered the following positive general comments about the site:

• “Keep up the good work. I have never had any major issues using the site.” • “I think you do an excellent job and I have enjoyed the support that I receive both on the

web and when I have emailed in the past. I have to pay my own dues and I feel I get my money’s worth and more! Thank you.”

• “I use this website rather infrequently and find what I am looking for without difficulty.” • “This web site has improved every year.”

2

Major Usability Issues and Recommendations

On the other side, our study found many candidates for improvement in ala.org, some of them serious. All of the assessment methods found that, in many content areas — areas ranging from contact information to divisional information to policy information — the difficulty of locating information fell anywhere from moderate to extreme.

ALA website users expressed the following general concerns about the site:

• “What I would like to do is to be able to get to where I’m going more quickly.” • “I sometimes find looking for articles cumbersome.” • “It is not intuitive at all.” • “I can rarely find what I’m looking for.” • “Badly in need of a re-design for better ease of use.” • “Librarians should be able to do better than this.”

The list below outlines the major issues, in rough order of descending severity. (Severity will be addressed in the forthcoming Strategy Recommendations Report.)

• Information architecture inconsistent with users’ mental models • Difficult and confusing navigation • Inadequate handling of member/professional/public content • Difficult and confusing search function • Poor visual/page design • Incorrect or outdated content and links • Cryptic terminology • Long, cryptic URLs

The remainder of this document describes and illustrates the major issues, lists some user comments related to them, and gives some recommendations for approaching them.

Note: The issue of server reliability caused problems in the usability testing and was mentioned by numerous survey respondents. However, this report does not address it as a major issue because it is not a usability design issue, and in any case ALA is aware of the problem and is working actively to correct it.

Information Architecture Inconsistent with Users’ Mental Models The information architecture of ala.org is complicated and often does not match users’ mental models of the information structure and where content should be. Examples include the following:

• The site’s content seems to be organized more closely around the ALA organizational structure than around the content topics. This causes difficulty in finding information, especially for people who are not familiar with how the Association is organized. For example, the book awards list is categorized by which component of ALA gives the

3

award. (Note: This problem is not limited to the content that is categorized under Divisions, Offices, and RoundTables.)

• Multiple pages have the same heading (e.g., “Site Map”). • Many pages are long lists of items, not well categorized (if at all).

Users offered the following comments:

• “The organization of ALA into Divisions should be clear and prominent (rather than having divisions listed alphabetically after committees!!) (See the ALCTS division page with its sections in oval across the top). Division activities should be integrated so everything a division does (committee, continuing ed offering) should have an automatic link from the ALA page on that subject. One should not need to know ALA's organization in order to find something! On the other hand, the divisions should be promoted and easy to find when people want to go to the division site.”

• “The site is so huge with so many areas to explore. It is hard to remember what are discussion groups, what are chapters and what are divisions. It would be nice to have a way to find gateways to all resources/etc. for youth services or for academic libraries in one place. Keep the categories that are there now, but add a thematic interface so that I can find grants, reading lists, new releases, divisions, discussion groups, etc. for one area of interest. (This would be like having subject and keyword fields in a MARC record.)”

• “The logic I use to determine where I think items I want to view are located doesn't seem to agree with the actual structure of the web site.”

• “The website is not technically difficult. It is badly organized and much information is hidden at very low levels. You have to know the entire organization hierarchy to find a lot of information because it is only accessed down the hierarchical arrangement.”

• “Items for which I am looking are rarely under the tabs which I think I will find them.” • “It is not readily apparent under which headings to look for information.” • “I'm not always sure whether what I'm looking for is listed under a division, chapter, etc.

So I pretty much only use the site for [division] guidelines, my committee activities, and to register for conferences.”

• “The divisions and groups within ALA are labyrinthian—if I were a new librarian I would find it very hard to locate the division and subdivision I would want to investigate.”

Following “Professional Tools” to “Intellectual Freedom” gives policy information, not tools.

4

• “It takes too many clicks to get to the divisions. They should not be buried so deeply into the site.”

• “I think you need to know what a "Division" is before you can get to the specific information many users want.”

• “Deep links make navigation difficult at times (having to click and click and click to get to where I want to go).”

• “The website design is unnecessarily complex.”

Recommendations:

• Organize information according to content. Except for the information specifically about ALA or the divisions, do not categorize the information — especially information of interest to the general public — according to ALA’s organizational structure. For example, categorize book awards by type of award (children’s books, etc.), not by what part of ALA awards them (Division/Office/etc.).

• Make the hierarchy “flatter” — i.e., increase the size of the levels and decrease their number. At each level, make more of the lower-level content directly reachable.

Difficult and Confusing Navigation Navigation on ala.org is often difficult and confusing, and is sometimes inconsistent. This issue surfaced in every assessment process we used — heuristic review, usability testing, online survey responses, and even the focus groups. One test participant who successfully reached the Intellectual Freedom toolkit stated, “I don’t even know how I got here. It is total serendipity.” Examples of this problem include the following:

• The left navigation area features an expandable menu with sub-items, and not all pages in the content areas are included as sub-items. This leaves the user wondering whether these pages can be reached via the menu and, if not, how they can be reached again.

• The website has no site map. The (multiple) pages called “Site Map” are actually part of a site index, and moreover are reached through a link labelled “Contact Us.”

• Links often lead to pages that are not what the link indicated. For example, the “Awards and Scholarships” button at the top leads, not to a top-level “Awards and Scholarships” page, but to the “Awards” page at the next level down from it.

• The breadcrumbs are sometimes incorrect. • Not all PDF files are so indicated. For example, the “Library and Information Studies

Education and Human Resource Utilization Statement” link on the Continuing Education page is a PDF but is not indicated as such.

5

This page is not shown in the list of contents of the “Membership” section. Where does it

The breadcrumbs imply (incorrectly) that “Human Resource Development and Recruitment” is a link on the home page.

These pages are called “Site Map” when they are actually an index.

The “Contact Us” link brings up this page, but the staff directory and most of the contact info are below the fold.

6


• “I had trouble finding some of the advocacy info I was looking for. It seems to me a lot of info is hidden inside other information and the search doesn't always take you there.”

• “I felt that I always need advanced search techniques to find what I needed. It was not necessarily intuitive.”

• “It would be nice if there was a drop-down menu or something to efficiently use the links along the top of the main site.”

• “A site map groups all pages into a format that is easily scanned. The alpha listing is cumbersome and doesn't give the user an idea where they would find their information topically.”

Recommendations:

• Use drop-downs and flyouts at upper levels, to help users navigate to lower levels without having to go through so many intermediate levels. (Note: Ensure that the top-level items of the flyouts are also links, so that the user is not required to use the flyout feature.)

• For long pages, use internal navigation. Provide a “page table of contents” at the top of the page, with links leading to top-level headings within the page.

• When listing links at the next level down, include in the list all pages at that next level. (Note: This means defining the levels so that the number of pages is manageable.)

• Whenever a link’s target is anything other than an image or HTML page (e.g., Word document, PDF file, etc.), indicate the type of file it is.

• Ensure that the breadcrumbs shown accurately reflect the position of the pages in the hierarchy.

Inadequate Handling of Member/Professional/Public Content The public content and the members-only content are generally not distinguished visually, except by the instruction to log in for access to members-only content. Problems in managing the user’s ALA membership arose in all data collection methods. Examples include the following:

• Clicking on “Join/Renew ALA” at the top of each page brings up a page with links for “Join ALA, Renew, or Add to Your Membership” — making it appear that those are three different links that will go to different places.

• Librarians who are not members complained that much of the information they needed was not available to them because it was in the members-only area.

• Clicking on “Join” in at least some of the Divisions’ sites takes the user to the ALA general “Join/Renew/Add” page without notifying them that they will be leaving the Division’s site.

• Logins are not persistent. A user may log in, but on later encountering further members-only content may be asked to log in again.

• Members complained about situations in which they were asked for a login and, once having provided it, were taken to a general page instead of the one they were trying to reach when they were asked for the login.

7

• ALA logins require members to know their membership numbers or have easy access to them.

• There is no visual distinction made between members-only pages and public pages, which can cause great confusion about where a user is because the content areas are all linked together so that users may go back and forth between them.


• “It seems that when I am in a hurry and need information from the ALA website, I do not have my log-in and password with me. It's frustrating! I think that some of the information that is secured behind the login/password could be placed in the pages open to the public. I feel that only my personal account information should be secured.”

• “When I want to log in I have to have my membership number. I don't have it with me both at home and work so I am limited in getting into the system. Is there a way it could remember my number from my IP addresses?”

• “The log-in was not 'consistent' about when and where to log-in and 'how long it lasted'.” • Once you get prompted to log in (which seems unnecessary to me) you get thrown out of

the site, so you have to follow the path back to it. Why can't you just log in and the get to the link you clicked on before the log-in prompt?

• “Some of the information I would like is only available to members, so I look for the info on other websites.”

• “I spent much too long trying to figure out how to edit my information – I could only easily find how to join or renew. I moved so I just want to change my address. There are WAY too many options and WAY too many words on the page. Could I just click a box marked ‘update my information?’”

Recommendations:

• Make the members-only and public content areas visually distinguishable in some way. (Note: They do not need to be very different, but the differentiator should be clear and noticeable.)

• Ensure that all information that is restricted to members really needs to be so restricted. • Separate the public content from the members-only content. (Note: It is not necessary,

or even advisable, to forbid member pages from linking to public pages, but it should be clear to which area a page belongs.)

• Provide publicly accessible professional information that is separate from members-only information, and limit the use of library jargon in content intended for the general public.

• Provide a specific logout function, and have logins persist until the visitor logs out.

These three links all go to the same page.

8

• Ensure that members, after logging in, are taken to where they were trying to go when they were asked for their login information.

• Allow members to choose their user IDs.

Difficult and Confusing Search Function Users almost universally found the search capability difficult. Examples of problems include the following:

• The searches often find a great deal of information that is outdated or is not very relevant to the query. This could well indicate a lack of adequate metadata.

• The search results appear under a large “Google” logo with another search box and a button that reads “Google Search.” A majority of the usability test participants mentioned that they were unsure whether the search had taken them outside of ala.org. Some participants even avoided the new search box and reverted to the original search box when searching for a different term. Although the search box that appears on the search results page has under it a pair of radio buttons labelled “public content” and “public and secure content,” these do not convey that what is being searched is within ala.org.

• The search results pages do not include “ALA” in the page titles, so it doesn’t show up in the browser window’s title bar.

• The results do not indicate the relevance of the items found.


• “I do not like your search capabilities. If I type in Newbery Award, the first or at least the first 3 choices should include the home page for the award, which does not happen. The search button in the online store is erratic, sometimes pulling up Read posters of celebrities and sometimes not.”

• “The search feature is usually not helpful.” • “I think the search engine needs to work better.” • “A better search engine (or perhaps better metadata?) might make searching more

successful.” • “The search feature doesn't yield precise results.” • “The search engine does not rank relevancy. So, a search for Newbery Award brings up

press releases from years ago, before a list of the overall Newbery winners.”

The Google logo’s prominence and the search button’s label cause confusion about the scope of the search.

9

Recommendations:

• Assess the metadata, and implement a policy for new and modified content, to ensure that indexing of the site’s pages works to support users’ search needs.

• Lessen the visual power of the Google logo on the search results page. Consider using a smaller “Powered by Google” logo, such as the following examples from other sites:

• Make it obvious that the search is within ala.org. • Consider providing a “sort by date” function for the search results. • Consider indicating the relevance of each entry in the search results. (Note: Some

studies have found that most people do not find relevance scores particularly meaningful or helpful, but it is entirely possible that librarians would.)

Poor Visual/Page Design The visual design is text heavy, and it generally strikes site visitors as outdated. Examples include the following:

• Many pages include unnecessary explanatory text at the top, text that interferes with users’ getting to the “meat” of the content. The focus groups commented on this problem on virtually every page we visited.

• The site uses too wide a range of font sizes. On most pages, the main font size is fairly large, the left-navigation links are medium-sized, and the links at the top of the page are small. When users adjust the text size in their browsers to make the small ones legible, the large ones become unwieldy and exacerbate the scrolling problem.

• Some pages have huge banners or logos at the top that occupy space that would be better used for content. For example, the banner in the 2007 Midwinter Meeting section is 252 pixels high, and the logo in the Human Resources Development and Recruitment section is 185 pixels high. The most common screen resolution is still 1024x768, and using one-third of screen real estate for decoration (as does the Conference section) is excessive.

• The “visited link” color is light and unsaturated, making visited links appear to be grayed out.

10


• “Clunky, too busy, too much hype and advertising - not enough meat.” • “Extremely TEXT HEAVY! Lose some of the text and add more buttons.” • “Too much text at the top of page.” • “The important info is beneath the fold.” • “The website design ... is extremely unprofessional in appearance. It's like it was

designed back in the 1990s before librarians were so internet-savvy, and no one has bothered to update it since then.”

• “The design is incoherent visually and spatially.” • “I think the ALA website could be set up to be easier to use. I know you have to have

some buttons at the top but if it could be set up more like a newspaper website with bold headings that are easy to find it may be easier to use.”

• “I do not like the font or perhaps it is the size of the font. The font size is too large.” • “The font sizes in the menu are good, but the very large font size used in the page text is

very difficult to read - I have to change my browser settings to display a smaller font size if I want to read more than a few sentences.”

• “When I print a page, the last inch or so is cut off. So, I have to highlight, cut and paste to a blank page in order to print and get the entire text.”

• “It makes pages print much longer than they should have to and when pages are printed they tend to cut off the sides. A redesign should use CSS to have a separate stylesheet applied when a document is printed so that it prints without all of the navigation and with a cleaner layout.”

Recommendations:

• For text that is to be read and used (other than headings and “fine print”), limit font sizes to between 0.9em and 1.1em.

• Limit banner heights to approximately 100 pixels, if possible.

This banner image is 252px high and occupies more than one-third of the screen’s real estate at 1024x768.

11

• Provide “tables of contents” at the tops of long pages, to let site visitors know what each page contains and provide a means for reaching its sections. (Some pages already have this feature.)

• Use an additional cascading style sheet designed specifically for printing the pages.

Incorrect or Outdated Content and Links Some of the information on the ALA site was found (or was considered by users) to be incorrect or obsolete. Examples include the following:

• The breadcrumbs are sometimes incorrect. For example, some of them imply that the page on which they appear is linked from the ALA home page, when it is not.


• “I found that committee rosters were out of date and that very little recent information was provided.”

• “Sometimes I get an official announcement about [a division] newsletter or ALA Washington Office tour, click on the URL and am told that that link no longer exists. This may be a problem with the sender but it has happened several times with different senders.”

Recommendations:

• Ensure that all content is up to date. For each content area, assign someone the responsibility to monitor this.

• Ensure that all links work. Consider regular use of an automated tool that checks for broken links within the site.

• Consider adopting a practice of, when a page is removed, leaving a placeholder page there for a while, with metadata that tells robots to stop indexing it.

Cryptic Terminology Some of the terminology used on the site is cryptic or full of jargon. Examples include the following:

• The Divisions page does not spell out the acronyms and initialisms for the Divisions. This may be acceptable for members who are very familiar with ALA and its components, but for new people or those who know only a few Divisions, it can be very confusing. Users commented about it, and it caused difficulties for some of the test participants in performing the task of getting information on joining one of the divisions.

• Some survey respondents criticized the use of library jargon on the site.

12


• “When I see an acronym for the first time, I expect it to be defined.” • “Categories seem to be erudite and excluding to average user with little time to waste

deciphering patronizing library lingo.”

Recommendations:

• Define or display the full name of any acronym when it occurs. • Examine the public content for unnecessary use of library jargon. (Note: A clearer

separation of the public, professional, and members-only contents may help.)

Long, Meaningless URLs Many of the URLs are very long and are neither understandable nor amenable to being copied and pasted. Examples include the following:

• http://www.ala.org/al_onlineTemplate.cfm?Section=American_Libraries&template=/ContentManagement/ContentDisplay.cfm&ContentID=143450

• http://ala.org/Template.cfm?Section=News&template=/ContentManagement/ContentDisplay.cfm&ContentID=143636

• Even fixed URLs may be longer than appropriate for usability — e.g., http://www.ala.org/ala/ProductsandPublications/Products_and_Publications.htm


• “The web addresses for inner pages should be smaller so links can be sent easier.” • “Links on individual pages are way too long, if someone wants to include them on other

sites, in publications, etc it is very cumbersome and unwieldy—and goes against other web norms.”

• “Web addresses are very long and cumbersome.” • “/Section=long-urls&suck=yes” (Yes, this is an exact quote.)

This list of Divisions, as undefined acronyms, means nothing to the novice user or to someone without prior knowledge of ALA’s structure.

13

Recommendations:

• Make URLs meaningful and reasonably short. • Review the situations that use CMS-generated, meaningless URLs. Eliminate them

where possible and reduce their use where they cannot be eliminated. • Shorten persistent URLs. For instance, in the example above,

“Products_and_Publications” could probably be abbreviated to “Prods_Pubs.”

14

Online Survey - Quantitative Analysis

Introduction The online survey asked respondents about what goals they had come to do, how many of them they achieved, what problems they had with the site, whether they used a library regularly, how often they visit ala.org, whether it has features they found particularly useful, and whether they use another website because the ALA site doesn’t meet their needs. We also administered a standard satisfaction questionnaire called the System Usability Scale, that we had adapted for ala.org.

In the month that the survey was on line, approximately 1200 people completed it. UserWorks conducted a quantitative analysis of all 1200 responses and a qualitative analysis of 200 responses chosen to represent the appropriate proportions of members and nonmembers; and from the qualitative analysis we eliminated responses from people whose only stated purpose for visiting the site that day was to fill out the survey. This section presents the results of the quantitative analysis that address usability. The qualitative results aimed at usability evaluation are reflected in the Major Issues section above, and the results aimed at users’ needs will be presented separately, in the User and Task Analysis Report.

User Satisfaction Satisfaction is the comfort and acceptability of use that a system or product provides its users, and can be measured as a set of subjective responses to a questionnaire. The ala.org evaluation used the System Usability Scale (SUS), a standardized usability/satisfaction questionnaire, at the end of the power user interviews and usability tests.

SUS scores range from 0 to 100, higher scores being associated with greater satisfaction. According to Dr. Robert Bailey, who has used SUS on many projects, average satisfaction scores usually fall between 65 and 70 (http://www.usability.gov/pubs/030106news.html). Normative data for SUS scores have not yet been established, however, and in general the studies that report SUS scores do so to compare products against one another rather than to assess one product against an expected level of satisfaction. The SUS will be useful for ala.org assessment in that it can enable the measurement of changes in user satisfaction—increases, it is hoped—as the site evolves.

Survey respondents’ SUS scores for ala.org ranged from 0 to 100 and had a mean of 53.1, below what is considered average. There were no significant differences between members and nonmembers on user satisfaction.

Quantitative Analysis The quantitative analysis consisted primarily of computing correlations between the survey questions aimed at satisfaction. For the question on how often they visit ala.org, we used the Spearman correlation, because that variable is ordinal. For the other items we used Pearson correlations, as the general consensus among measurement specialists is that a Likert scale can be treated as an interval variable.

The following table presents the correlations among the quantitative variables assessed. Most of the variables were correlated at the P<.001 level, the exceptions being the frequency of library

15

use and the frequency of visits to ala.org. Not all of the correlations are meaningful, however; for example, the correlation between use of bricks-and-mortar libraries and agreeing with the statement that ala.org has useful features was .09, which is very weak but was statistically significant because the number of respondents was so large.

Not surprisingly, the satisfaction score was positively correlated with how well the respondents felt their goals had been met on that visit to ala.org and with their level of agreement with the statement that the site has useful features, and negatively correlated with their level of agreement with the statement that they go to other sites for some purposes because ala.org doesn’t adequately support them. Less strong were the correlations between goals achieved on this visit and their tendency to use other sites. Frequency of visiting ala.org and physical libraries did not show any meaning correlations with any satisfaction variables.

Goals

Achieved ALA

Useful Other Site

ala.org Visits

Library Use

SUS Score Corr. Sig.

.448 <.001

.563 <.001

-.589 <.001

.060 n.s.

-.007 n.s.

Goal(s) achieved on this visit

Corr. Sig.

.357 <.001

-.330 <.001

.040 n.s.

.047 n.s.

ALA site has useful features

Corr. Sig.

-.409 <.001

.119 <.001

.094 <.001

Use other site because ALA.org doesn’t meet needs

Corr. Sig.

.023 n.s.

.005 n.s.

Frequency of visits to ALA.org

Corr. Sig.

.003 n.s.

16

Accessibility Compliance

Accessibility Evaluation Methods UserWorks conducted an accessibility review to determine which elements of the American Libraries Association web site does not meet the accessibility guidelines set forth by W3C and to make recommendations on how to make the Web site more accessible to users with disabilities. The accessibility review employed a number of different approaches, including browsing the site using an automated accessibility evaluation software tool and manually checking the site for specific accessibility problems.

We conducted a basic accessibility review of the ALA Web site, examining how readily it could be used by people with disabilities. We checked the site for compliance with W3C’s priority-1 and -2 guidelines. We began by using Bobby, the automated testing tool for accessibility checking, to support the evaluator in locating and categorizing systemic problems. Bobby was not used to identify specific accessibility issues, but rather to identify areas of the site that should be manually reviewed to determine whether an actual accessibility problem existed. Bobby was the ideal tool for a site as extensive as ALA’s. �

ALA has a very content-rich and extensive site. This resulted in only a sampling of the pages used for the accessibility evaluation and not the entire site. We sampled 30,000 links and their associated web pages in the accessibility review. In addition, we looked at the site with JavaScript both on and off.

Test Environment The accessibility evaluation was conducted mainly on computers running Windows XP. We also tested it on a computer running Mac OS X, 10.4.8. The following table lists the hardware and software configurations that were used to test the ALA Web site:

Feature Windows Macintosh

Processor Speed: Pentium IV Intel Core Duo

Memory: 1 GB RAM 1 GB RAM

Internet Connection

Cable and DSL Cable

Operating system Windows XP Professional service pack 2

Mac OS X Tiger, 10.4.8

Browsers tested Internet Explorer, Mozilla Firefox

Mozilla Firefox, Safari

Screen resolution 1024 x 768 1024 x 768

Accessibility technology

Bobby, manual review Manual review

17

Accessibility Scorecard This list of guidelines is grouped by priority, as defined by the World Wide Web Consortium. The definitions of the priorities are as follows:

Priority 1 - A Web content developer must satisfy this checkpoint. Otherwise, one or more groups will find it impossible to access information in the document. Satisfying this checkpoint is a basic requirement for some groups to be able to use Web documents.

Priority 2 - A Web content developer should satisfy this checkpoint. Otherwise, one or more groups will find it difficult to access information in the document. Satisfying this checkpoint will remove significant barriers to accessing Web documents.

The following table shows the broad scope of how well ala.org meets W3C’s WAI checkpoints. Note that the site is very large and we did not test all of its pages. The table lists results for the pages that we did test.

The only column that is definitive is a “No” answer, because it means that we observed a violation of the checkpoint. A “Yes” means that all of the pages we tested complied with the checkpoint. “N/A” means that the checkpoint requires certain features in certain circumstances, and the ALA website does not fall under those circumstances (and so is not required to have those features). “N/O” also means that the checkpoint requires certain features under certain circumstances, and one of three conditions holds: (1) we are not sure whether any pages in ala.org fall under those circumstances, (2) some do and we did not observe any violations in the relatively small number of relevant pages we assessed, or (3) the checkpoint is a programmatic one (e.g., 6.2, below) and is out of scope of this review.

Priority 1 checkpoints

In General (Priority 1) Yes No N/A N/O

1.1 Provide a text equivalent for every non-text element (e.g., via "alt", "longdesc", or in element content). This includes: images, graphical representations of text (including symbols), image map regions, animations (e.g., animated GIFs), applets and programmatic objects, ascii art, frames, scripts, images used as list bullets, spacers, graphical buttons, sounds (played with or without user interaction), stand-alone audio files, audio tracks of video, and video.

X

2.1 Ensure that all information conveyed with color is also available without color, for example from context or markup. X

4.1 Clearly identify changes in the natural language of a document's text and any text equivalents (e.g., captions). X

18

6.1 Organize documents so they may be read without style sheets. For example, when an HTML document is rendered without associated style sheets, it must still be possible to read the document.

X

6.2 Ensure that equivalents for dynamic content are updated when the dynamic content changes. X

7.1 Until user agents allow users to control flickering, avoid causing the screen to flicker. X

14.1 Use the clearest and simplest language appropriate for a site's content. X

And if you use images and image maps (Priority 1) Yes No N/A N/O

1.2 Provide redundant text links for each active region of a server-side image map. X

9.1 Provide client-side image maps instead of server-side image maps except where the regions cannot be defined with an available geometric shape.

X

And if you use tables (Priority 1) Yes No N/A N/O

5.1 For data tables, identify row and column headers. X

5.2 For data tables that have two or more logical levels of row or column headers, use markup to associate data cells and header cells.

X

And if you use frames (Priority 1) Yes No N/A N/O

12.1 Title each frame to facilitate frame identification and navigation. X

And if you use applets and scripts (Priority 1) Yes No N/A N/O

6.3 Ensure that pages are usable when scripts, applets, or other programmatic objects are turned off or not supported. If this is not possible, provide equivalent information on an alternative accessible page.

X

And if you use multimedia (Priority 1) Yes No N/A N/O

1.3 Until user agents can automatically read aloud the text equivalent of a visual track, provide an auditory description of the important information of the visual track of a multimedia presentation.

X

19

1.4 For any time-based multimedia presentation (e.g., a movie or animation), synchronize equivalent alternatives (e.g., captions or auditory descriptions of the visual track) with the presentation.

X

And if all else fails (Priority 1) Yes No N/A N/O

11.4 If, after best efforts, you cannot create an accessible page, provide a link to an alternative page that uses W3C technologies, is accessible, has equivalent information (or functionality), and is updated as often as the inaccessible (original) page.

X

Priority 2 checkpoints

In General (Priority 2) Yes No N/A N/O

2.2 Ensure that foreground and background color combinations provide sufficient contrast when viewed by someone having color deficits or when viewed on a black and white screen. [Priority 2 for images, Priority 3 for text].

X

3.1 When an appropriate markup language exists, use markup rather than images to convey information. X

3.2 Create documents that validate to published formal grammars. X

3.3 Use style sheets to control layout and presentation. X

3.4 Use relative rather than absolute units in markup language attribute values and style sheet property values. X

3.5 Use header elements to convey document structure and use them according to specification. X

3.6 Mark up lists and list items properly. X

3.7 Mark up quotations. Do not use quotation markup for formatting effects such as indentation. X

6.5 Ensure that dynamic content is accessible or provide an alternative presentation or page. X

7.2 Until user agents allow users to control blinking, avoid causing content to blink (i.e., change presentation at a regular rate, such as turning on and off).

X

7.4 Until user agents provide the ability to stop the refresh, do not create periodically auto-refreshing pages. X

7.5 Until user agents provide the ability to stop auto-redirect, do not use markup to redirect pages automatically. Instead, configure the server to perform redirects.

X

20

10.1 Until user agents allow users to turn off spawned windows, do not cause pop-ups or other windows to appear and do not change the current window without informing the user.

X

11.1 Use W3C technologies when they are available and appropriate for a task and use the latest versions when supported. X

11.2 Avoid deprecated features of W3C technologies. X

12.3 Divide large blocks of information into more manageable groups where natural and appropriate. X

13.1 Clearly identify the target of each link. X

13.2 Provide metadata to add semantic information to pages and sites. X

13.3 Provide information about the general layout of a site (e.g., a site map or table of contents). X

13.4 Use navigation mechanisms in a consistent manner. X

And if you use tables (Priority 2) Yes No N/A N/O

5.3 Do not use tables for layout unless the table makes sense when linearized. Otherwise, if the table does not make sense, provide an alternative equivalent (which may be a linearized version).

X

5.4 If a table is used for layout, do not use any structural markup for the purpose of visual formatting. X

And if you use frames (Priority 2) Yes No N/A N/O

12.2 Describe the purpose of frames and how frames relate to each other if it is not obvious by frame titles alone. X

And if you use forms (Priority 2) Yes No N/A N/O

10.2 Until user agents support explicit associations between labels and form controls, for all form controls with implicitly associated labels, ensure that the label is properly positioned.

X

12.4 Associate labels explicitly with their controls. X

And if you use applets and scripts (Priority 2) Yes No N/A N/O

6.4 For scripts and applets, ensure that event handlers are input device-independent. X

7.3 Until user agents allow users to freeze moving content, avoid movement in pages. X

21

8.1 Make programmatic elements such as scripts and applets directly accessible or compatible with assistive technologies [Priority 1 if functionality is important and not presented elsewhere, otherwise Priority 2.]

X

9.2 Ensure that any element that has its own interface can be operated in a device-independent manner. X

9.3 For scripts, specify logical event handlers rather than device-dependent event handlers. X

Positive Notes We found most of the pages to be fairly accessible. Although Bobby cited many potential violations of the W3C guidelines, through manual review we found that most of these instances were well within the range of accessibility. We found that web pages and their links are accessible with scripts turned off. The layout and organization of information also follow W3C guidelines. The next sections will discuss the areas of the web site that did not meet W3C accessibility standards.

Accessibility Issues This section presents the aspects of the website that violated W3C priority 1 and 2 guidelines for accessibility that were manually discovered after the use of Bobby. There are also examples shown with each violation.

Priority 1 Violations

(Checkpoint 1.1) Provide alternative text for all images:

A significant portion of the graphics do not contain alt attributes. When an image does not have an alt attribute, screen readers will often read the name of the file instead. This makes navigation difficult for visually impaired users, because they have to listen to the filenames of numerous layout and presentation related graphics before getting to the real content. Some alt attributes for graphics used as links describe what the graphic looks like, rather than describing where the link leads.

All graphics should contain alt attributes. Graphics used for layout purposes should contain blank alt attributes (alt=""). The alt attributes for graphics used as links should describe where the link leads, rather than the content of the graphic. For graphic links that have an accompanying text link, the alt attribute for the graphic should be blank. This review found missing alt text in numerous files, including the following

http://www.ala.org/presstemplate.cfm?Section=presscenter

http://www.ala.org/Template.cfm?Section=awards&template=/ContentManagement/ContentDisplay.cfm&ContentID=140592

http://www.ala.org/PrinterTemplate.cfm?Section=issues&Template=/ContentManagement/HTMLDisplay.cfm&ContentID=141177

https://www.ala.org/BookLinksTemplate.cfm?Section=booklinks

22

http://www.ala.org/Template.cfm?Section=bookmediaawards&template=/ContentManagement/ContentDisplay.cfm&ContentID=140592 (see screen shot, below)

(Checkpoint 1.1) Provide transcripts for all audio files: (or captions for all video files)

Many of the video and audio media on the website lack accompanying transcripts or captions for these types of files. Visually or audibly impaired users will have trouble using this type of content and will not be able to access the information that is presented in these ways. These media types should have transcripts and captions accompanying them.

http://www.ala.org/Template.cfm?Section=presskits&template=/ContentManagement/ContentDisplay.cfm&ContentID=127572

Priority 2 Violations

(Checkpoint 13.1) Create link phrases that make sense when read out of context:

There are many links on the website that do not include any context within the link itself. For instance, the “go!” button (see below) may cause confusion for a user with a screen reader. Although the alt attribute does label that button as “submit” the context may not be specific

This image has no alt tag, so IE for Windows displays “Image”.

23

enough for that particular use and a user may not be able to glean the function of that button. In addition, the term “submit” is computer centric; “Search” would be much better.

http://www.ala.org/al_onlineTemplate.cfm?Section=2006contents&Template=/ContentManagement/ContentDisplay.cfm&ContentID=114838

(Checkpoint 3.1) Where it's possible to mark up content instead of using images, use a markup language:

There are many graphics within the ALA website that are used to represent text. It would be easier for users with visual disabilities to have these messages available in text instead, primarily so that the font size will enlarge with the rest of the text on the page. Additionally, many of these same images do not offer enough information in their alt attributes.

http://www.ala.org/Template.cfm?Section=librariesandyou

Browser Review The assessment found no differences in the accessibility of ala.org among the browser/OS combinations. However, there were a few differences when the ALA website is viewed in different browsers. In Firefox and Safari, some of the menu options do not line up as they are intended and do not appear as they do in Internet Explorer. Additionally, the link text for the site map does not appear on the screen, but if users click in that area it will still bring up the linked page. Although there are some visual differences between the browsers, there were no differences in function or accessibility.

24

Microsoft Internet Explorer

Mozilla Firefox

25

Apple Safari

26

Appendix A – ALA Member Participant Screener

Directions Sent: Recruiter: Reminder Call:

Study: American Library Association (ALA) Web Site Administrators: Jimmy and Dustin

October 16-20, 2006 Number of participants needed: 6

UserWorks’ Office; 90 minutes (schedule 2 hours apart) $75

ALA Usability Test Screener – MEMBERS

Name:

Male Female

Daytime Phone #

Evening Phone #

Points for Recruiter to cover:

• American Library Association’s web site

• Looking at what members want from it

• How they use it, what they think of it, how they would group the information

• 90 minutes

Just to confirm: Are you a member of the American Library Association?

Yes

No (refer to Nonmembers screener)

Are you a librarian? Yes No

If no: What is your occupation?

1. If you work in a library, what kind of library is it?

Public library

School library

College or research library

Specialized library

Other (specify: )

27

Which of the following categories includes your age? (Recruit a mix)

Under 18 (Terminate)

18-29

30-39

40-49

50-59

60+

2. Excluding email and instant messages, approximately how many hours per week do you use the Web?

0 (Terminate)

1 to 10 hours

11 to 25 hours

26 hours or more

3. Do you use ALA’s Web site?

Yes – how often?

No (if never, terminate)

4. The session will be video recorded. Only the team working on this project will use the recording, and your name will not be associated with the recording or other data in any way. You will be asked to sign an informed consent form. Would you be willing to be video recorded?

Yes

No (Terminate)

5. How would you like the directions to UserWorks sent to you?

Email Address

Fax Number

Over Phone

If you are not able to keep this appointment, please call me at 301.431.0500 as soon as possible.

28

Appendix B – ALA Nonmember Participant Screener

Directions Sent: Recruiter: Reminder Call:

Study: American Library Association (ALA) Web Site Administrators: Jimmy and Dustin

October 16-20, 2006 Number of participants needed: 6

UserWorks’ Office; 90 minutes (schedule 2 hours apart) $75

ALA Usability Test Screener – NONMEMBERS

Name:

Male Female

Daytime Phone #

Evening Phone #

2 Librarians who are not members of ALA, mix of library types

2 college students

2 members of the general public who use a physical library at least twice a year

Points for Recruiter to cover:

• Web site of a nonprofit professional organization

• Looking at how nonmembers might use the site, what they think of it, how they would group its information

• 90 minutes

6. What is your occupation?

If Librarian or library employee:

Are you a member of any professional organizations?

Yes – which one(s)? (if ALA member, refer to Members screener)

No

29

7. How frequently do you use a library? (any kind of library)

Do not use (Terminate)

Fewer than 10 times a year (Terminate)

10-20 times a year

21-50 times a year

More than 50 times a year

8. Which of the following categories includes your age? (Recruit a mix)

Under 18 (Terminate)

18-29

30-39

40-49

50-59

60-69

70+

9. What is the highest level of education you’ve completed? (recruit a mix)

Less than high school

High School

Some college

Associate’s degree or trade school

Bachelor’s degree

Master’s or professional degree

PhD

Other:

10. Excluding email and instant messages, approximately how many hours per week do you use the Web?

0 (Terminate)

1 to 10 hours

11 to 25 hours

26 hours or more

30

11. The session will be video recorded. Only the team working on this project will use the recording, and your name will not be associated with the recording or other data in any way. You will be asked to sign an informed consent form. Would you be willing to be video recorded?

Yes

No (Terminate)

12. How would you like the directions to our office to be sent to you?

Email Address

Fax Number

Over Phone

If you are not able to keep this appointment, please call me at 301.431.0500 as soon as possible.

31

Appendix C – Informed Consent Form

I, _________________________________________, freely and voluntarily consent to participate in this usability study under the direction of UserWorks, Inc.

I understand that my participation is completely voluntary and that I may withdraw my consent and discontinue my participation at any time without penalty or prejudice to my business organization or me.

I authorize UserWorks, Inc. to use the findings from this evaluation, with the provision that my name will not be associated with any of the results, for release to the developers of this product.

I have been given the right to ask questions concerning the procedures to be employed during this study and to have these procedures explained to my satisfaction.

Video Recording Release Video recordings made during this study will be used for research and development. I have been informed that my work during this evaluation will be recorded and viewed by the staff of UserWorks, Inc. and their client.

I give my consent to UserWorks, Inc. and their client to use my recorded image and voice for these purposes, with the provision that my name will not be associated with the recording.

I have read and understood the foregoing and understand that I may receive a copy of this form on the day of the study.

Participant’s Name: _________________________________________

Participant’s Signature: _________________________________________

Date: _______________________________

32

Appendix D – Participant Demographics

Member Age Occupation Library Visits Internet hrs/wk

Laura 18-29 grad student weekly 11-25

Oksana 30-39 Sr. Cat. Lib. Tech occasionally 1-10

Marybeth 50-59 school librarian occasionally 11-25

Pat 50-59 Private org periodically 26+

Marina 50-59 specialized librarian occasionally 1-10

Galima 50-59 specialized librarian weekly 11-25

Nonmember Per Year

Jace 30-39 Pastor/Doctoral student 50+ 11-25

Kathryn 40-49 Pediatric Occ. Therapist 21-50 11-25

Diana 40-49 specialized librarian 26+ 26+

Adrian 60-69 retired 21-50 11-25

Susi 50-59 teacher/phd student 21-50 11-25

Inna 40-49 librarian 50+ 11-25

Serge 50-59 librarian 50+ 11-25

33

Appendix E – Test Administrator Guide

(Refresh browser; clear out any persisting data entered from previous sessions)

Pre-test Interview Questions

How frequently do you visit your local Library?

Have you ever used online library services?

Are you a member/familiar with the American Library Association?

Have you ever been to ALA.org? What for? How long ago?

Orientation

Before we begin working with the ALA web site, I want you to remember a few things:

• We’re being videotaped. That’s so I can go back and review your comments if I need to.

• Your name will not be associated with the recording, and your name won’t appear in our report.

• Please remember that I am not testing you. You are helping us test how well this web site design communicates information and meets your needs as a potential user. If you have trouble answering a question or locating the information, the problem is not you, it suggests a need to improve the web site.

• As you think about how you would interact with this web site, I would like you to THINK ALOUD.

• I’d like you to comment on both POSITIVE and NEGATIVE aspects of the design. What you like and don’t like. What you think would work well, or not work so well.

• As we go through various tasks, please indicate to me when you think you are DONE. Also indicate if there are situations where you would seek HELP and how you would go about that.

• There will be parts where you need to enter a name and address information, you can just make up examples.

Do you have any questions before we begin?

(Have the participant go through one of the two sets of tasks, depending on audience type.)

(Read each task to the participants, one at a time. Reread the tasks as needed.)

34

Members

Task 1: Register for an Event

You have been asked to attend the ALA 2007 midwinter conference in Seattle.

Find out how much the conference will cost and how to register.

Find the price for a hotel to stay in for the duration of the conference.

Return to the home page.

Task 2: Divisional Info

You are interested in joining ALA’s Reference and User Services Association and would like information on what benefits come with your membership. Find out the cost and how to join.

Benefits with membership?

Cost of membership?

How to register?


Task 3: Intellectual Freedom

You are interested in Intellectual Freedom and have heard that the ALA has toolkit that might be useful to you regarding this topic.

Locate this toolkit on the website.


Task 4: Find Contact Info

You need to contact someone who specializes in Government Relations. A colleague suggested that you contact Erin Haggerty, and told you that her contact info is on the website.

Can you find an email address and phone number for her.

Return to the home page

35

Task 5: Policy Statement

You would like to know ALA’s policy on Intellectual Freedom and its policy on membership.

Find information about ALA’s policy on Intellectual Freedom.

Find information about ALA’s policy on Membership.


Task 6: Publications

You are interested in publications offered by ALA.

Find information about how to purchase celebrity READ posters.


Task 7: Grants

You are looking for a grant to help support research for your doctoral dissertation.

Find out if ALA offers research grants.

Find an appropriate grant for doctoral students.


Task 8: Standards

You are looking for the ALA standards manual to help you understand ALA’s procedures.

Find ALA’s standards for Authority of Divisions.


Nonmembers

Task 1: Calendar of ALA Events or Programs

You would like to know what ALA sponsored programs or events will be occurring over the next month or two.

36

Find out what types of programs/events ALA sponsors, if any.

Find out if there are any events/programs happening over the next two months in your area.


Task 2: Membership Info

You are interested in becoming an ALA member. Find out the benefits and costs associated with becoming a member.

Find out the ALA member benefits.

Find out how much it would cost to join ALA.

Find out how to join.


Task 3: Literacy Programs/Statistics

You are interested in adult literacy. Find out some related information about this topic.

Find out what the adult literacy rates are for this country.

Investigate if there are any literacy programs sponsored by ALA.


Task 4: Intellectual Freedom

You are interested in Intellectual Freedom and censorship issues.

Find out ALA’s main issues regarding Intellectual freedom.

Find out what you can do to support ALA in this issue.


Task 5: Legislative News

You are interested in the most recent legislative news regarding libraries and the ALA.

37

Find out which legislative issues the ALA reports on.

Find out the most recent legislation regarding the PATRIOT ACT.


Task 6: Award Winners

You would like to find some award winning book titles.

Find out the most recent winners or the Caldecott and Newbery book awards.

Find out the title of book that won the 1992 Newbery award.


Task 7: Advocacy

You are interested in library advocacy and would like to know how you can become an ALA advocate.

Find out what issues the ALA advocates.

Find out how to become an ALA advocate.


Task 8: Purpose of ALA

You would like to find information regarding ALA’s purpose and the ways it supports that purpose.

Find out ALA’s stated purpose.

Find out how ALA supports that purpose.


38

Appendix F – Usability Testing Notes

Introduction UserWorks conducted a usability test of the current ALA Web site (ala.org) in October, 2006. Thirteen representative users, drawn from two different target audience groups, interacted with the Web site, using tasks that focus groups had previously identified as the most important tasks for each audience group. UserWorks observed the extent to which these participants could perform the tasks, and we recorded the navigational paths that each participant took for each task. We identified usability issues, if any, associated with each task. The information contained in these notes will form a substantial part of the forthcoming Usability Assessment Report.

Objectives This usability test had four main objectives:

• Determine the extent to which the site allows users to successfully complete a series of tasks deemed to be important to each specific user group

• Measure the site’s ability to meet the information needs of representative users

• Assess the effectiveness of the site’s communication with its user

• Highlight design issues of the web site.

Methods

Participants

Thirteen participants took part in this usability test. We recruited six ALA members and seven ALA nonmembers for individually conducted test sessions. Of the nonmembers, three worked as librarians. The members had varying degrees of library use, internet use, and familiarity with ALA.

All participants — eleven women and two men — had experience in finding information on the web. All lived in the greater Washington, DC area and were paid incentives of $75 each for participating in sessions, which lasted approximately one hour.

Participants who were not ALA members were recruited from UserWorks’ database of volunteers, based on criteria that we had specified in conjunction with ALA. Members were recruited from a list of local members provided by ALA. Candidate participants responded with an expression of interest to an initial email announcement about the study and then were interviewed by telephone using the screening questionnaire for their audience group (Appendix A, Appendix B). All participants signed the informed consent form (Appendix C).

Appendix D summarizes the demographics of the participants in this study.

Facilities and equipment

All sessions were conducted in person at UserWorks’ usability lab in Silver Spring, Maryland. Test sessions used UserWorks’ in-house usability lab equipment, which permitted simultaneous real-time recording of the screen with which the participant was interacting and a superimposed

39

video camera image of the participant’s face. All sessions were digitally recorded, with capture of full-screen resolution.

Participants accessed ala.org over a high-speed DSL connection. They viewed the site on a 17-inch monitor with the resolution set at 1024 x 768. The site was live and working off the current server.

Test procedures

The test administrator greeted the participants and gave them a brief orientation and overview. Before showing them the web site, he asked them some questions about their library habits, electronic library experience, and level of familiarity with ALA and its Web site. He then asked them to open the browser and go to the ALA home page, and he encouraged them to think aloud as they worked through a series of predetermined and user-group-specific tasks. Each task posed a scenario that asked them to find a particular piece of information on the site. The tasks used in the testing had been identified as the six most important tasks for ALA members and nonmembers, defined during focus groups that UserWorks had previously conducted with members of those two user groups.

The test administrator carried on a running dialog with the participants to find out how well the site was matching their expectations and supporting their information needs. He avoided coaching the participants or offering cues, and allowed them to determine when they had completed the tasks. Participants were asked to comment on both positive and negative aspects of the site, as well as any ideas they had for improvement. At the end of the test sessions, the administrator interviewed participants about their experience with the site, asking specifically about their overall impressions of the site.

Analysis and reporting

The usability data collected in the test sessions consisted of observational notes on participant behavior in interacting with the site, the navigational path or click stream of each participant per task, and germane comments made by the participants. An observer took notes on a laptop computer. Participant behavior and comments were categorized by task. For each task, the level of success/failure and pages viewed were compared across participants and user groups. The video recordings were reviewed as needed to clarify specific incidents or comments.

These usability test notes divide the findings into two groups, those for ALA members and those for nonmembers. It breaks the findings for each group by task, describing each task and presenting the performance and issues related to the tasks.

Findings

Members

Task 1 – Mid-winter conference

You have been asked to attend the ALA 2007 midwinter conference in Seattle.

• Find out how much the conference will cost and how to register.

• Find the price for a hotel to stay in for the duration of the conference.

40

Performance: All but one of the members were able to complete this goal without error. Four of the six participants selected the “Events & Conferences” menu from the top navigation. Of note, only one participant followed the link in the main area of the home page. Once on the conference page, the participants fared fairly well in finding the appropriate rates for the conference and methods of registering. The subtask of finding hotel prices for the conference caused a few more problems, as not all participants noticed the top menu. One participant noted her confusion that the menu items on the top did not replicate the items on the left. One participant failed to notice the top menu at all. We observed one participant completing the task, although through the unusual path of starting in “Our Association.”

Issues: This task is a very important one for members of ALA, and it is encouraging that participants fared well with it. The biggest issue was that the conference page violated its own information architecture by neglecting to offer all submenus and related topics in the left column navigation.

Task 2 – Divisional Information

You are interested in joining ALA’s Reference and User Services Association and would like information on what benefits come with your membership. Find out the cost and how to join.

• Benefits with membership?

• Cost of membership?

• How to register?

Performance: Four of the six participants successfully completed this task. Two failed it outright; both of them ran into problems once they tried to join or add to their membership. They got to the RUSA page but became confused once they selected “join RUSA” and were taken back to the ALA home page. One of the successful participants commented that the task was more difficult than it needed to be. An issue that was discussed here was the use of unfamiliar acronyms. One participant stated that “when I see an acronym for the first time, I expect it to be defined.” The routes taken to access this information were consistent. Four of the members went directly to “Our Association.”

Issues: Two main issues came out of this task: The use of undefined acronyms makes finding an appropriate division more difficult than necessary; and users face confusion when having to decipher the relationship between a division and its parent (RUSA and ALA). Some users either failed to notice or took some time to notice that to join the division they had to add to their ALA membership, from the ALA home page.

Task 3 – Intellectual Freedom

You are interested in Intellectual Freedom and have heard that the ALA has a toolkit that might be useful to you regarding this topic.

• Locate this toolkit on the website.

Performance: Two of the participants found the toolkit with relative ease and felt good about their results. Both of these people used the search box to find the toolkits. Half of the participants went directly to “Professional Tools” and then used the left-side link to “Intellectual Freedom.” One of the successful participants even started there before using the search box. Quite a few

41

comments were made that participants were surprised not to find the toolkits under “Professional Tools.” One user who got to the toolkit stated, “I don’t even know how I got here. It is total serendipity.”

Issues: The fact that people struggled with this task suggests that the toolkits are located in a place that does not make sense to users. As on participant put it, “When you mentioned intellectual freedom and tools, for me it was logical to go to “Professional tools.””

Task 4 – Contact Info

You need to contact someone who specializes in Government Relations. A colleague suggested that you contact Erin Haggerty, and told you that her contact info is on the website.

• Can you find an email address and phone number for her.

Performance: Participants generally struggled with this task. Two were able to find the information from the staff directory. Four of the participants failed the task. Of interest, two of the participants who failed this task actually reached the “Contact Us” page but failed to find the contact details after searching under her last name and the office where she works. By presenting the Site Map under the “Contact Us” link, the site confuses users with inconsistent labeling.

Issues: This task presented several different issues. For one, the very top navigational items were commented to be “too small” and “hard to see.” Of the users who clicked the “Contact Us” link, half missed the “A-Z Staff Directory” — which was not only listed under A but was below the fold. These people’s behavior suggested that they believed the main A-Z listing was the Staff directory. Also, participants who entered the contact’s name into the search bar were not directed to her contact details in the directory within the first page of results.

Task 5 – Policy

You would like to know ALA’s policy on Intellectual Freedom and its policy on membership.

• Find information about ALA’s policy on Intellectual Freedom.

• Find information about ALA’s policy on Membership.

Performance: Participants displayed a wide range of techniques for this task. The most common technique was to use the search bar; after that, people searched through the top headings with no particular consistency. Although no one failed the task outright, only two participants were able to complete the task with no trouble. In general this task created a lot of searching under various headings. Some participants commented that they were surprised that this information was so hard to find, especially because of “how much importance ALA places on policy.”

Issues: One of the big issues from this task came from participants’ using the search box. To the user, the policy information did not seem to fit logically under any of the main headings, which prompted users to use the search box. The issues came once the search results were presented: The search results appear under a large “Google” logo with another search box and an action button that reads “Google Search.” A majority of the test participants mentioned that they were unsure whether the search had taken them outside of ala.org. Some participants even avoided the new search box and reverted to the original search box when searching for a different term. The other obvious issue here is that users to do not see a reasonable place for policy information to go.

42

Task 6 – Publications

You are interested in publications offered by ALA.

• Find information about how to purchase celebrity READ posters.

Performance: Participants fared well on this task. Not a single participant failed this task, although not everyone completed it with 100% success. In two cases, participants were able to purchase a poster from the site; however, they failed to find the celebrity posters as directed. In general, participants started by going either to the ALA store or to “Productions & Publications.”

Issues: The online store is well laid out, and users had good success finding what they wanted once they were in the store. However, getting to the store from “Products & Publications” created some problems.

Task 7 – Grants


• Find out if ALA offers research grants.

• Find an appropriate grant for doctoral students.

Performance: Half of the participants went straight to “Awards & Scholarships” for this task. Two people were able to find the appropriate grants while others found the list of grants.

Issues: Just about everyone, successful or not, stated that they would have more success if the grants were broken down into different categories. One remarked, “When I am searching for (grants) I don’t care if it’s from a division or office. They could break down [the type] on the left.” Another issue involved the server’s responding slowly and crashing. This was a consistent problem during the days of testing.

Task 8 – Standards


• Find out if ALA offers research grants.

• Find an appropriate grant for doctoral students.

Performance: Participants did well with this task. Eighty percent found the information easily without error. Participants used a variety of paths to reach the manual. The majority of successful participants used the search box, one found it through the “Contact Us” A-Z listing under S, and another found it under “Professional Tools” then “Standards and Guidelines.”

Issues: None. The most important finding from this task is that it can be done via a variety of paths. This is a good feature and should be modeled throughout the rest of the site.

Task 9 – Literacy


• Find out what the adult literacy rates are for this country.

• Investigate if there are any literacy programs sponsored by ALA.

43

Performance: Only three participants had enough time to attempt this task, because of the time they used in the previous tasks. Two of the three failed to find a literacy rate with which they were confident; one was able to extract that info from a PDF document. All three participants tried the search box, and two of the participants stated that they would not use this site for such information. When it came to finding literacy programs, all three participants found the Office for Literacy and Outreach Services (OLOS) internal pages but only one found an appropriate program to satisfy this task.

Issues: One issue that came out of this task is the site’s lack of providing quick and easy factual information on certain topics. Participants mentioned in other parts that they liked such a feature and exclaimed that it would be useful with literacy issues. The OLOS page has a huge amount of extra resources but no way for the user quickly to find one of interest. Another issue with this task was that the language used did not work well for participants. Participants who clicked on “Programs and Events” (within OLOS) were surprised to find conference events and failed to find the “Outreach Resources” which contained more appropriate items. Also, no one found the “Literacy Directory” listed further down the page. The last issue that came up here, as well as in other sections, was the size of font used on the site. Participants said that although the font was easy to read, it took up too much space on the page, thus forcing the user to scroll.

Task 10 – Awards


• Find out the most recent winners or the Caldecott and Newbery book awards.

• Find out the title of book that won the 1992 Newbery award.

Performance: Three participants reached this task (for reasons of time). All three were able to complete this task with without error. One participant followed the “Awards & Scholarships” heading, and the remaining two used the search box. Users were split on their opinions of this task. One thought it was easy and “user friendly”; one was impartial, as she was able to find what she wanted relatively quickly using the search; and the third stated that “the navigation was clumsy and clunky.” The last expanded on that, saying that there was “too much content on the center of the page (under “Awards & Scholarships”) and that moved everything down.” He stated that he would have preferred ways to narrow his search on that page.

Nonmembers

Task 1 – Events

You would like to know what ALA sponsored programs or events will be occurring over the next month or two.

• Find out what types of programs/events ALA sponsors, if any.

• Find out if there are any events/programs happening over the next two months in your area.

Performance: Participants fared well with this task, on the whole, although two struggled with it. No participants failed this task and most were able to complete the task without any problem. All but one of the participants went to the heading “Events & Conferences” from the home page.

44

Issues: The two participants who struggled with this task both became confused with the categorization of events. First off, participants became confused when they found past events listed under the upcoming events heading. Secondly, participants commented that the sorting features seemed to be both inadequate and backwards. The “sort by date” feature did not give a sorting by date, but the calendar under “sort by group” did. One participant commented about the “sort by date” section, saying that “it said ‘sort by date’ but this has been sorted alphabetically.” Also, multiple people stated that they would like a “sort by region” option. This is most likely an artifact from our task but a reasonable request nonetheless. When participants were able to find the events calendar, they seemed pleased with content but again a little confused with the inclusion of past events. This is probably because participants usually reached the calendar through upcoming events, and then sorted by group.

Task 2 – Membership

You are interested in becoming an ALA member. Find out the benefits and costs associated with becoming a member.

• Find out the ALA member benefits.

• Find out how much it would cost to join ALA.

• Find out how to join.

Performance: Overall, participants did well with this task. All but one completed the task with little or no difficulty. Five of the seven participants went directly to “Our Association” and then chose the “Membership” link from the left navigation. Of interest, only one participant clicked the “Join/Renew” link from the top of the home page.

Issues: The first issue from this task is that users thought the placement of membership under “Our Association” was good. The “Join/Renew” link at the top of the page was said to be unreadable by many. Some stated that it was too small; others added that the italicized “giveALA” is almost illegible. The one participant who struggled with this task became confused when reading the requirements of membership. He believed that he was ineligible to join even though he had worked as a librarian for 20 years. He said that the requirements needed a better explanation. Users did become confused by having to scroll down to content appropriate to their level of membership.

Task 3 – Literacy


• Find out what the adult literacy rates are for this country.

• Investigate if there are any literacy programs sponsored by ALA.

Performance: This task was split by participants. Three failed the task and three succeeded while one succeeded but did so with difficulty. Five of the participants used the search box for this task.

Issues: The three people who failed this task moved all around the site without finding a lead. One stated “none of these shout out ‘literacy’ to me.” He found OLOS but became frustrated that the info was not quickly accessible. “If I can’t find the answer in 30 seconds I become frustrated and would go somewhere else.” One participant found an “ALA Fast Facts” link and commented

45

that the literacy pages could do with one. Two of the users who had difficulty with this task had found the OLOS page but found its content difficult to grasp.

Task 4 – Intellectual freedom

You are interested in Intellectual Freedom and censorship issues.

• Find out ALA’s main issues regarding Intellectual freedom.

• Find out what you can do to support ALA in this issue.

Performance: Participants struggled with this task. Two were able to complete the task without any problems, and two completed the task with some difficulty. (All but one started out by selecting the “Issues & Advocacy” heading from the home page.) Two of the participants were unable to complete the task because the server was not responding. Two participants were unable to find a list of related issues within the Intellectual Freedom information pages.

Issues: The biggest issue with this task is that the main points of interest regarding Intellectual Freedom are not easily found. There is a lot of information on Intellectual Freedom on this site; however to get to the main sections, the user must click on “Federal Legislative Issues” within “Our Association.” This requires the user to have pre-existing knowledge or to get lucky. Another point that was highlighted here is that the breadcrumbs do not actually show the topic’s location within the site. More server problems highlighted the back-end issue once again.

Task 5 – Legislative Issues

You are interested in the most recent legislative news regarding libraries and the ALA.

• Find out which legislative issues the ALA reports on.

• Find out the most recent legislation regarding the PATRIOT ACT.

Performance: No one failed this task outright; two finished with some difficulty. (Two could not finish because of server problems.) Three participants successfully completed this task. All but one participant started this task by selecting the main heading “News.” Of those four, three of the participants then chose to use the search box, after not finding the “News” section relevant.

Issues: Most participants went directly to “News” but then left that section because they did not find what they were looking for. One participant commented that the “Press Center” page is what he expected when he clicked “News.” (He had been taken to the “Press Center” by selecting the heading “News” under the “Advocacy Resource Center.”) The issue here is that the “News” page contains content that the participants were not expecting. Too much informative or directional text appears at the top of the page.

Task 6 – Awards


• Find out the most recent winners or the Caldecott and Newbery book awards.

• Find out the title of book that won the 1992 Newbery award.

46

Performance: Participants were generally successful with this task. Every participant used the “Awards & Scholarships” heading. Every participant had success with this task, and only one had any difficulty due to site design. Again, the server created problems.

Issues: The confusion regarding this task came from the categorization of awards into divisions and offices, etc., and when the links took the user to an external site.

Task 7 – Advocacy

You are interested in library advocacy and would like to know how you can become an ALA advocate.

• Find out what issues the ALA advocates.

• Find out how to become an ALA advocate.

Performance: Server problems prevented two of the participants from completing this task. All other participants had success with this task and all of them started by selecting the “Issues & Advocacy” heading.

Issues: The amount of information provided on the topic of Advocacy is extensive. This page handles the info better than other parts because it offers categories for that information within the center content as well as a “Getting Started” link.

Task 8 –Purpose

You would like to find information regarding ALA’s purpose and the ways it supports that purpose.

• Find out ALA’s stated purpose.

• Find out how ALA supports that purpose.

Performance: Four participants attempted this task (the others had run out of time); all of them completed the task successfully. Three of the participants read the mission statement from the central content under “Our Association.” One read it from the home page.

Issues: None

usability assessment report: american library association website

Documents