Annotated Bibliography 1 Greene & Moore April 5, 2013

Annotated Bibliography Courtney Greene & Kate B Moore From: One Search to Rule Them All: Mapping the Literature on User Experience and

Discovery, ACRL 2013 Poster Session

Arcolio, Arnold, and Felicia Poe. 2008. “WorldCat Local at the University of California Usability

Testing: Round One, Spring 2008.” http://libraries.universityofcalifornia.edu/about/oclc_docs/WCL_Summary_03Sept2008.pdf.

Usability testing at University of California found that WorldCat Local was generally preferred by faculty and graduate students over the traditional catalog, and the inclusion of journal content and the ability to search for resources from other libraries were seen as especially useful. However, participants had problems with identifying and accessing online resource available to them, and a misunderstanding of how the relevancy ranking in WorldCat Local worked (e.g. WCL ranked items held at the local institution higher than items of higher relevancy at other institutions).

Asher, Andrew D., Lynda M. Duke, and Suzanne Wilson. 2012. “Paths of Discovery: Comparing the

Search Effectiveness of EBSCO Discovery Service, Summon, Google Scholar, and Conventional Library Resources.” College & Research Libraries (April 15). http://crl.acrl.org/content/early/2012/05/07/crl­374.full.pdf+html.

This comparative study of EBSCO Discovery Service, Summon, Google Scholar and ‘no specified tool’ observed students from two institutions completing research tasks, and then reported on librarian evaluation of results found by participants. Significant findings included: that EDS had some statistically significant edge over the others in terms of performance and quality of results found, mostly to do with relevancy ranking; and that students rarely if ever go past the first page of results.

Awre, Christopher, Richard A. Green, Simon W. Lamb, and Diane Leeson. 2010. “Blacklight at Hull.”

https://hydra.hull.ac.uk/catalog/hull:3451.

The University of Hull in the UK undertook two usability studies of Blacklight in 2010—the first asked users to review Stanford University and University of Virginia’s instances of Blacklight. Results from the first usability test found that basic searching and getting the full record to display were the easiest actions, while refining searches and finding items in a particular library were the most difficult. The second study asked users to review the University of Hull’s instance of Blacklight, and results from the second usability test confirmed those of the first, with users easily completing basic searches, but finding it difficult to refine searches through facets. However, in both studies, once the user saw and used the facets, subsequent use was generally

One Search To Rule Them All: Mapping the Literature on User Experience & Discovery ACRL 2013 | Indianapolis IN | Poster Session

Annotated Bibliography 2 Greene & Moore April 5, 2013

easier. User comments on searching and the layout of Blacklight interfaces were generally positive. One drawback noted in the second round of testing was that in some instances, the relevancy ranking of results led to what one user termed, “a loss of precision” (18). While relevancy ranking can be adjusted locally, the University of Hull noted that these adjustments required a significant amount of time and effort at other institutions.

Ballard, Terry. 2011. “Comparison of User Search Behaviors with Classic Online Catalogs and

Discovery Platforms.” Charleston Advisor 12 (3): 65–66. doi:10.5260/chara.12.3.65.

Using Google Analytics, and thorough analysis of search logs, this study found that users were fifteen times more likely to refine searches in a discovery interface (Encore) than in the library’s (traditional) Innovative Interfaces WebPac catalog (65).

Bauer, Kathleen. 2008. “Yale University Library VuFind Test—Undergraduates.”

http://tinyurl.com/3wvvgpd.

Yale University Library’s 2008 user study of VuFind, early in the product’s lifecycle, seems to have been influential, with others adopting its tasks for use in their own published studies. Participants in the study were able to find known items, but had difficulty narrowing their searches using topical facets like author and subject. Bauer noted that relevancy ranking for VuFind needed improvement, and that students were confused about some of the labels used in basic search, including “All fields.”

Becher, Melissa, and Kari Schmidt. 2011. “Taking Discovery Systems for a Test Drive.” Journal of Web Librarianship 5 (3): 199–219. doi:10.1080/19322909.2011.583134.

Becher and Schmidt report on testing conducted at American University comparing local implementations of AquaBrowser and WorldCat Local. Half of their users preferred WorldCat Local, while 30% had no preference; however, graduate students were more likely to either prefer AquaBrowser or have no preference. When asked why users preferred WorldCat Local, many respondents noted its simple interface. American University eventually chose Summon as their discovery layer because it was seen as having the features and functionality found by users as desirable in AquaBrowser and WorldCat Local, including a simple but academic interface, facets, and links to full­text through an openURL link resolver.

Bertot, John Carlo, Katy Berube, Peter Devereaux, Kerry Dhakal, Stephen Powers, and Jennie Ray. 2012. “Assessing the Usability of WorldCat Local: Findings and Considerations.” Library Quarterly 82 (2): 207–221.

This study worked with participants from an information science degree program (editorial note: library students and thus likely inclined to be somewhat more oriented to librarian thinking in the opinion of this reviewer). Testers had generally positive reactions to interface but struggled with advanced search, which they found to be cluttered. Other confusions or frustrations included the

One Search To Rule Them All: Mapping the Literature on User Experience & Discovery ACRL 2013 | Indianapolis IN | Poster Session

Annotated Bibliography 3 Greene & Moore April 5, 2013

need to log in repeatedly from off campus to access items; and that labels of formats were confusing (visual material confused for video clip). Some preferred to search library website. The biggest issues identified related to resource location, and figuring out how to find and obtain items, which was to some degree specific to WCL and to the specific library. Generally it was felt that WCL was stronger with known item searches than with topic or exploratory searches.

Bietila, David and Tod A. Olson. 2012. “Designing an Evaluation Process for Resource Discovery

Tools.” In Planning and Implementing Resource Discovery Tools in Academic Libraries, ed. Diane Dallis and Mary Pagliero Popp, 122–136. Hershey, PA: IGI Global. doi:10.4018/978­1­4666­1821­3.ch008.

This chapter provides a rubric for testing the usability for discovery layers by type of use, including known item searching, subject searching, narrowing results through facets, and finding full­text. Usability testing was conducted on EBSCO Discovery Service, and comments were generally positive. Findings suggested that relevancy ranking and an intuitive interface were extremely important, as none of the participants went beyond the first page of results, and no one used the help screens available.

Boock, Michael, Faye Chadwell, and Terry Reese. 2009. “WorldCat Local Task Force: Report to

LAMP”. Oregon State University. http://hdl.handle.net/1957/11167.

Usability testing with faculty, students, and staff at Oregon State University found that WorldCat Local (WCL) was generally preferred over the traditional catalog, as was the case at University of California. Although participants indicated that WCL and the catalog were equally adept at known item searching, actual searches in the catalog for known items produced better results. As with the usability studies at University of California (Arcolio and Poe 2008), integration of articles and resources from other libraries in the search results was deemed to be very useful by students and faculty. Other features about WorldCat Local that Oregon State University users liked were images of book jackets, ratings, faceted searching, and the ability to export citations.

Chapman, Suzanne, Scott Dennis, Kathleen Folger, and Ken Varnum. 2012. "Developing a

User­Centered Article Discovery Environment." In Planning and Implementing Resource Discovery Tools in Academic Libraries, ed. Diane Dallis and Mary Pagliero Popp, 194­224. Hershey, PA: IGI Global. doi:10.4018/978­1­4666­1821­3.ch012. This chapter describes the user­centered research methods undertaken by the University of Michigan in order to choose a discovery layer, and later, to test user satisfaction of the system chosen, Serials Solutions Summon. Methods included a persona analysis to analyze the needs of different groups of users, a survey asking participants to rate Google Scholar, the federated search engine MetaLib, and Serials Solutions Summon based on twelve features, and guerrilla usability testing that focused on quality, comprehensiveness, and effectiveness of the interface. Later follow up surveys found that users preferred Summon to Google Scholar and MetaLib for finding the full­text of articles online, advanced searching, and limiting searches to scholarly

One Search To Rule Them All: Mapping the Literature on User Experience & Discovery ACRL 2013 | Indianapolis IN | Poster Session

Annotated Bibliography 4 Greene & Moore April 5, 2013

materials only. Comeaux, David J. 2012. “Usability Testing of a Web­Scale Discovery System at an Academic

Library.” College & Undergraduate Libraries 19 (2­4): 189–206. This test spanned many user groups ­ undergraduates, graduate students and faculty ­ and was

specifically interested in interactions with facets and with overall usability. Amalgamation of results into one big set was a concern, and this proved harder for faculty than for undergrads, (although the latter were also little confused). The system was deemed very learnable based on improvements in performance throughout the tests. Existing difficulties with location labels were made more visible by the discovery tool; this was fixed by correcting underlying data. In one task, issues of scope of tool (suppressing items not held in collection) were exacerbated by terminology on screen (searching ‘everything’). Other issues discussed centered on library practices, phraseology, etc.

Dallis, Diane, and Mary Pagliero Popp, eds. 2012. Planning and Implementing Resource Discovery

Tools in Academic Libraries. Hershey, PA: IGI Global. doi:10.4018/978­1­4666­1821­3.

As the title indicates, this book provides a thorough review of academic libraries’ decision­making processes for reviewing and choosing a discovery layer for their institution. There are numerous chapters discussing usability studies of specific discovery layers; specific chapters detailing these studies are covered elsewhere in the bibliography.

Dartmouth College Library. 2009. “An Evaluation of Serials Solutions Summon As a Discovery

Service for the Dartmouth College Library.” http://www.dartmouth.edu/~library/admin/docs/Summon_Report.pdf.

Dartmouth College Library, a Summon beta partner, conducted an array of user testing methods including surveys, interviews, and “remote viewing” sessions from September 2008 to December 2009 and found that undergraduates, faculty and library staff generally viewed the product positively, as fast, simple, and “familiar,” but that graduate students compared it unfavorably to subject­specific databases. Specific “unmet discovery needs” for all groups included cited reference searching, known author searches, and provision of the user’s search history.

Denton, William, and Sarah J. Coysh. 2011. “Usability Testing of VuFind at an Academic Library.”

Library Hi Tech 29 (2): 301–319. doi:10.1108/07378831111138189.

Denton and Coysh, while receiving positive feedback on the interface from test participants, pointed to a need for improvement both in the default relevancy functionality and in VuFind’s inability to provide a left­anchored title browse.

Emanuel, Jennifer. 2011. “Usability of the VuFind Next­Generation Online Catalog.” Information

One Search To Rule Them All: Mapping the Literature on User Experience & Discovery ACRL 2013 | Indianapolis IN | Poster Session

Annotated Bibliography 5 Greene & Moore April 5, 2013

Technology & Libraries 30 (1): 44–52. doi: 10.6017/ital.v30i1.3044.

A study conducted at the University of Illinois at Urbana­Champaign (UIUC) showed that patrons from all user groups had a generally positive opinion of VuFind, in most cases preferring it to Amazon when asked to choose between the two. Recommended changes to VuFind as a result of the usability study included integration with RefWorks, simplification of the “favorites” listing feature, and better linking to other libraries in the shared Illinois catalog I­Share.

Fagan, Jody Condit. 2010. “Usability Studies of Faceted Browsing: A Literature Review.” Information

Technology & Libraries 29 (2): 58–66. doi: 10.6017/ital.v29i2.3144.

Based on the literature reviewed, this article proposes best practices for evaluating faceted library interfaces with the goal of better study design overall, cautioning, “when faceted browsing is applied to a real­life situation, implementation details can greatly affect user performance and user preferences” (63). Specifically, Fagan recommends striving for greater consistency and specificity across the board, perhaps through developing a shared base protocol for library user testing of faceted systems, so that results might be better able to be compared amongst libraries. A 2010 literature review on usability studies of faceted searching revealed that use of facets in library systems was generally well­received by participants during user testing of several different discovery interfaces.

Fagan, Jody Condit, Meris A. Mandernach, Carl S. Nelson, Jonathan R. Paulo, and Grover Saunders.

2012. “Usability Test Results for a Discovery Tool in an Academic Library.” Information Technology and Libraries 31 (1): doi:10.6017/ital.v31i1.1855.

Fagan et al published the results of a usability test of EBSCO Discovery Service (EDS) undertaken at James Madison University (JMU) in late fall 2010, consisting of nine tasks, and pre­ and post­test questions. They noted, like Williams and Foster, that participants frequently made use of facets and limiters, and largely ignored the Integrated Search functionality – JMU has since removed it, and instead provides links to library content (research guides, etc). Difficulties arose in tasks requiring participants to limit by format where labeling was not intuitive (“non­print resources” rather than “video”) and in a task involving specific holdings of a journal which required users to interpret link resolver results. The ten participants (including two faculty members) were also asked to complete the System Usability Scale, a standard scale that can be used for benchmarking user satisfaction across a variety of systems and applications, following completion of all tasks . EDS scored 56 out of a possible 100, an improvement from its 1

SUS results from spring 2010, in which it scored 37; the authors also tested the traditional library catalog (SUS score: 68) and an instance of VuFind (SUS score: 80) and noted “discovery tools, which support more tasks [than a library catalog], must make compromises in usability that simpler systems can avoid” (103).

1 See Brooke, John. 1996. “SUS ­ A Quick and Dirty Usability Scale.” In Usability Evaluation in Industry, edited by P.W. Jordanet. London: Taylor and Francis. http://www.usabilitynet.net/trump/documents/Suschapt.doc

One Search To Rule Them All: Mapping the Literature on User Experience & Discovery ACRL 2013 | Indianapolis IN | Poster Session

Annotated Bibliography 6 Greene & Moore April 5, 2013

Fahey, Sue, Shannon Gordon, and Crystal Rose. 2011. “Seeing Double at Memorial University: Two

WorldCat Local Usability Studies.” Partnership: The Canadian Journal of Library & Information Practice & Research 6 (2): 1–14.

Fahey et. al. conducted two separate usability studies on WorldCat Local (WCL), one with health and sciences students and faculty, and another with arts and sciences students. Findings reported from these studies only include tasks undertaken during both studies (searching for journals and books); therefore findings related searching for articles are not included. The study found that students from both discipline areas experienced many of the same problems, including locating the correct edition of a book, finding call numbers for print books, and verifying holdings information for print journals in WCL.

Foster, Anita K. and Sarah C. Williams. 2012. "Early Adoption: EBSCO Discovery Service at Illinois

State University." In Planning and Implementing Resource Discovery Tools in Academic Libraries, ed. Diane Dallis and Mary Pagliero Popp, 488­498. Hershey, PA: IGI Global. doi:10.4018/978­1­4666­1821­3.ch028.

Usability testing at Illinois State University after implementing EBSCO Discovery Service found that while participants frequently used facets on the basic search page (with the peer­reviewed facet most often used), other facets were rarely used, including those on the results page. The authors note that feedback received indicated that though students were able to quickly and easily use the basic search, that more instruction would be beneficial for using advanced features.

Freund, LeiLani, Christian Poehlmann and Colleen Seale. 2012. “From Metasearching to Discovery:

The University of Florida Experience.” In Planning and Implementing Resource Discovery Tools in Academic Libraries, ed. Diane Dallis and Mary Pagliero Popp, 22­43. Hershey, PA: IGI Global. doi:10.4018/978­1­4666­1821­3.ch002.

Discusses usability testing at the University of Florida on the metasearch engine MetaLib as a forerunner to discovery layer products. Usability testing on MetaLib found that participants had problems with the terminology used (including the name MetaSearch), understanding format icons, and finding coverage dates for journals. The University of Florida eventually decided to implement Serials Solutions Summon after usability testing among faculty and staff found that 60% of respondents preferred it to the other systems, primarily because of its interface and underlying data.

Fyn, Amy F., Vera Lux, and Robert J. Snyder. 2013. “Reflections on Teaching and Tweaking a

Discovery Layer.” Reference Services Review 41 (1): 113–124. doi:10.1108/00907321311300929.

This usability study was undertaken at Bowling Green State University and looked into testing

One Search To Rule Them All: Mapping the Literature on User Experience & Discovery ACRL 2013 | Indianapolis IN | Poster Session

Annotated Bibliography 7 Greene & Moore April 5, 2013

three of the customizations made to the library’s instance of Summon, including newspapers appearing in search results automatically, limiting initial search to items held at the library, and providing recommendations of subject databases to continue research. While an overwhelming number of the undergraduate respondents indicated their interest in the database recommender (though it was unclear whether students would actually use it), results for the other two customizations were divided, with almost half of respondents indicating that their preference for the exclusion of newspapers in initial search results, and an almost 50/50 split on whether items not held at the library should be included.

Gallaway, Teri Oaks, and Mary Finnan Hines. 2012. “Competitive Usability and the Catalogue: A

Process for Justification and Selection of a Next­Generation Catalogue or Web­Scale Discovery System.” Library Trends 61 (1): 173–185.

This paper is primarily concerned with documenting how analysis of transaction logs and discussions with library staff led to identifying areas of difficulty in using the library catalog; tasks for user testing of several candidate web scale systems were developed based on that data, but no specific information linking the issues with any named systems. That said, it is useful reading for big picture or preparatory work but does not report back user data per se.

Greene, Courtney. 2012. “The Long and Winding Road: Implementing Discovery at Indiana University

Bloomington Libraries.” In Planning and Implementing Resource Discovery Tools in Academic Libraries, ed. Diane Dallis and Mary Pagliero Popp, 499–514. Hershey, PA: IGI Global. doi:10.4018/978­1­4666­1821­3.ch029.

The Indiana University Bloomington (IUB) Libraries participated as an EDS beta partner and was one of five libraries across North America at which EBSCO did on­site testing of an early version of the EDS product. IUB implemented EDS, branded as “OneSearch@IU,” in August 2011, following a year­long implementation; this unusually long implementation period was attributed primarily to local constraints related to transitions in staffing, and to ongoing work in processing their SIRSIDynix catalog extract to ensure optimum display.

Gross, Julia, and Lutie Sheridan. 2011. “Web Scale Discovery: The User Experience.” New Library

World 112 (5/6): 236–247. doi:10.1108/03074801111136275.

Librarians at Edith Cowan University in Perth, Australia published the results of a usability study of Summon which found that a redesign of the library home page to feature a Summon search box “achieved the aim of simplifying navigation and steering students in one direction” (242). Like other researchers testing other web scale discovery tools, Gross and Sheridan observed difficulties arising from participants’ confusion in distinguishing between formats, which were exacerbated by inconsistent labeling – in their case, e­reserve items proved very difficult to find, due to their being labeled “web resource” in the results list. They also noted that participants were persistent in their use of the tool “even when an alternative pathway may have proved more fruitful” (244). Finally, use of limiters and of the advanced search screen was very limited,

One Search To Rule Them All: Mapping the Literature on User Experience & Discovery ACRL 2013 | Indianapolis IN | Poster Session

Annotated Bibliography 8 Greene & Moore April 5, 2013

practiced by only one of five participants. Hagedorn, Kat, Michael Kargela, Youn Noh, and David Newman. 2011. “A New Way to Find: Testing

the Use of Clustering Topics in Digital Libraries.” D­Lib Magazine 17 (9/10): 1–10. This paper focused around testing topic modelling using HathiTrust data in a test interface built on VuFind, the goal of which was to develop an algorithm that looks for patterns in full text then reports back topic terms to the user. Unmoderated testing of over three hundred users, the majority of whom were undergraduates, occurred at the University of Michigan and at Yale University. When participants were asked to use any facet to limit search, 73% chose the topic facet; but when specifically asked to use topic facet about a fifth didn’t. Satisfaction ratings of items found decreased by ½ to ⅓ with use of topic facet, but when using ‘all facets’ satisfaction was high. This was followed by moderated testing which compared algorithm topic facet with LCSH ­ they were rated roughly even by participants. Participants were more pleased overall when using topic facet in conjunction with other facets (that is to say, the topic facet was useful as a refinement) and there was some evidence of its being useful for serendipitous finding.

Hessel, Heather, and Janet Fransen. 2012. “Resource Discovery: Comparative Results on Two

Catalog Interfaces.” Information Technology and Libraries 31 (2): 21–44.

This article provides additional information concerning the usability study undertaken at University of Minnesota, Twin Cities in late 2009, which compared user reactions to the traditional catalog from ALEPH (MNCAT Classic) and their implementation of Primo (MNCat Plus), both Ex Libris products (See Primo Management Group. 2010. “University of Minnesota Digital Conservancy: MNCAT Plus and MNCAT Classic Survey : Results and Analysis.” http://conservancy.umn.edu/handle/92473).

Ho, Birong, Keith Kelley, and Scott Garrison. 2009. “Implementing VuFind as an Alternative to

Voyager’s WebVoyage Interface: One Library’s Experience.” Library Hi Tech 27 (1): 82–92. doi:10.1108/07378830910942946.

It is interesting to note that log analysis of the VuFind installation at Western Michigan University showed that Library 2.0 features such as tagging, commenting, and other social networking­type functions were little used; perhaps more unexpectedly, facets were used in only about a third of basic searches.

Holman, Lucy, Elias Darraj, Jonathan Glaser, Alice Hom, Heather Mathieson, Deane Nettles, and

Aronya Waller. 2012. "How Users Approach Discovery Tools." In Planning and Implementing Resource Discovery Tools in Academic Libraries, ed. Diane Dallis and Mary Pagliero Popp, 252­267. Hershey, PA: IGI Global. doi:10.4018/978­1­4666­1821­3.ch014.

Reports on usability testing at the University of Maryland of EBSCO Discovery Service, Ex Libris’ Primo, and Serials Solutions’ Summon based on four main areas: interface, search features,

One Search To Rule Them All: Mapping the Literature on User Experience & Discovery ACRL 2013 | Indianapolis IN | Poster Session

Annotated Bibliography 9 Greene & Moore April 5, 2013

facets, and the ability to save items found for future research. A summary of findings for each discovery tool is available on page 260. Problems found with all discovery layers tested included the use of library jargon, problems finding non­scholarly materials (e.g. textbooks, audiovisual); the most profound problem found for all of the discovery layers tested was the lack of relevant results based on terms used for topical searching.

Jarrett, Kylie. 2012. “FindIt@Flinders: User Experiences of the Primo Discovery Search Solution.”

Australian Academic & Research Libraries 43 (4): 278–299.

This study recruited undergraduate students to test FindIt@Flinders (the branded Primo implementation) comparative to their Voyager OPAC interface. Overall, success ratings for the two interfaces differed by only 3% although the Voyager interface was rated slightly more difficult. Some difficulties found seem to be related to both labelling and placement of interface or facet items. Facets were heavily used.

Johnston, Nicole, Alicia Salaz, and Rob O’Connell. 2013. “Determining Usability of VuFind for Users

in the United Arab Emirates.” Code4Lib Journal (19): 1–12.

This article reports on using VuFind to display results from the library catalog and from Summon in a single screen with two columns at a university whose population are largely non­native English speakers. Participants largely ignored social features (although it should be noted that they weren’t specifically requested to use them); interestingly, faculty testers actively disliked that the features appeared, as their presence increased scrolling. Adjustments were later made to the interface to prioritize crucial information. There was some concern on the part of the authors regarding presenting information left­to­right (given that Arabic reads right­to­left) but users appeared to be comfortable with l­to­r in an English language tool. The study concluded that by and large the response to the new interface was positive. Other findings included problems figuring out how to access full­text, and an interesting discussion of the “emotional intelligence” of labels, that is to say, trying to find ways to say things that were more specific and directive which reduced patron irritation (a specific example was changing from MULTIPLE LOCATIONS to CLICK HERE FOR LOCATION INFO).

Klein, Michael B. 2010. “Hacking Summon.” Code4Lib Journal (11): 11–18.

Klein of Oregon State University wrote in Code4Lib Journal about ways in which OSU has leveraged the Summon API to improve and simplify display of library catalog records through consolidating library location codes, removing information from MARC records (stripping Library of Congress information from government documents classified using SuDOCS, for example) and providing enhanced availability information.

Majors, Rice. 2012. “Comparative User Experiences of Next­Generation Catalogue Interfaces.”

Library Trends 61 (1): 186–207.

One Search To Rule Them All: Mapping the Literature on User Experience & Discovery ACRL 2013 | Indianapolis IN | Poster Session

Annotated Bibliography 10 Greene & Moore April 5, 2013

In this comparative study, the author tested several web­scale interfaces with the same tasks. Overarching takeaways included: library terminology and practices confused users (named catalogs, ambiguity about types of sources [ie is a government document a book], multiple records for multiple manifestations); the expectation of a Google­like single search box and associated search behavior was prevalent; few users went beyond the first page of results; and generally testers liked the discovery model which presents many kinds of sources together, with a pattern of first conducting a broad search which then can later be narrowed. Other interesting overall notes included that Amazon has taught people to want to work with metadata (faceting); Share/email/select/mark/export functions are important; the necessity of providing context about what’s being searched; the need both for better alignment between systems and available library services to make it easy to get help. Constructive criticism offered by system included: EDS ­ association with EBSCO interface skewed expectations to articles, difficulty of getting to ILL from system; Encore ­ cluttered & busy interface, no advanced search; Primo ­ difficulty in finding and using options, difficulty of getting to ILL from system; Summon ­ hard to access email function, difficulty of getting to ILL from system, labeling/confusion about ‘add results beyond library collection’ (users did not understand label), problems with link resolver working well, search refinements reset frequently and this was frustrating to users; WorldCat Local­ hard to access email function, difficulty of getting to ILL from system, reduce scrolling to make it easier to parse detailed results, search refinements reset frequently and this was frustrating to users.

Mischo, William H., Mary C. Schlembach, Joshua Bishoff, and Elizabeth M. German. 2012. "User

Search Activities within an Academic Library Gateway: Implications for Web­scale Discovery Systems." In Planning and Implementing Resource Discovery Tools in Academic Libraries, ed. Diane Dallis and Mary Pagliero Popp, 153­173. Hershey, PA: IGI Global. doi:10.4018/978­1­4666­1821­3.ch010.

Using an analysis of transaction logs from their locally created federated search system, Mischo et al. found that known item searching accounted for almost 50% of searches and that the most prevalent type of search is whatever the system’s default is. Taken together with the finding that help screens are rarely used indicates the importance of “automated and adaptive system assistance” (163). They also found that the features most likely to be used in discovery layers were direct links (to full text), spell­check, and limiting searches to a phrase or title.

O’Hara, Lisa, Pat Nicholls, and Karen Keiller. 2012. "Search Success at the University of Manitoba

Libraries Pre­ and Post­Summon Implementation." In Planning and Implementing Resource Discovery Tools in Academic Libraries, ed. Diane Dallis and Mary Pagliero Popp, 268­287. Hershey, PA: IGI Global. doi:10.4018/978­1­4666­1821­3.ch015.

One Search To Rule Them All: Mapping the Literature on User Experience & Discovery ACRL 2013 | Indianapolis IN | Poster Session

Annotated Bibliography 11 Greene & Moore April 5, 2013

This chapter describes usability testing on Summon at the University of Manitoba. Participants generally liked Summon’s interface; though some noted that the facets and limiters needed to be more apparent. Only one user used the facets without prompting (and after using them, none clicked on the “more options” of each facet), and none of the users saw the ability to save search “refinements” for new searches. (It was interesting to note that it was the most inexperienced searcher who refined the search using facets without being prompted, but the researchers hypothesize that this may be due to other’s past experiences with the university’s previous systems.) Participants generally agreed their searching was successful, and found that being able to save items in a folder and email them, along with the ability to view abstracts in the results were the most useful features of the system.

OCLC. “Findings from WorldCat Local Usability Tests: July, 2009­September, 2010.” https://www.oclc.org/content/dam/usability­labs/213941usf_some_findings_about_worldcat_local_2011.pdf

OCLC has conducted and published its own usability studies on WorldCat Local, which have been used to refine and enhance its content and interface. Findings from this report provide insight into ambiguity surrounding terms used in labeling (e.g. “Internet resource,” database) and discussion of the need for “short, predictable, reliable path to content,” especially online content, which is imperative for user satisfaction with discovery layers (2).

Olson, Tod A. 2007. “Utility of a Faceted Catalog for Scholarly Research.” Library Hi Tech 25 (4):

550–561.

An early usability study on the use of facets by PhD students. Findings indicate that the facets in the discovery tool reviewed, AquaBrowser, were useful for finding multi­disciplinary research that was not discovered using University of Chicago’s traditional catalog.

Palsson, Felicia. 2012. "Usability Testing Summon on the USC Libraries Home Page." In Planning

and Implementing Resource Discovery Tools in Academic Libraries, ed. Diane Dallis and Mary Pagliero Popp, 303­318. Hershey, PA: IGI Global. doi: 10.4018/978­1­4666­1821­3.ch017.

Compares the usability of the University of Southern California Libraries’ website before and after implementing Summon as the default search on their homepage. It was found that using Summon improved the ability for users to find commonly searched items, including books, articles, and research guides in a particular discipline. Findings also confirm that users tend to search in whichever tab/field was open by default, though the majority of users did eventually explore the other tabs.

Powers, Amanda Clay. 2011. “EBSCO’s EDS: Relying on Patron Data To Show the Way.” Library

Journal 136 (Reference Supplement) (November 15): 14–15.

The Mississippi State University Libraries implemented EDS in August 2010, simultaneous to a

One Search To Rule Them All: Mapping the Literature on User Experience & Discovery ACRL 2013 | Indianapolis IN | Poster Session

Annotated Bibliography 12 Greene & Moore April 5, 2013

total redesign of the library website. They found that while EDS was popular with undergraduate students (as the library staff had expected) it was also very popular with faculty and doctoral students: “serendipity returned as cross­disciplinary searches became a reality with very narrow topics” (15). There was also an increase in questions about the library catalog ­ obtaining items through ILL from consortial libraries, etc.

Preater, Andrew. 2010. “Mental Models and User Experience of a Next­Generation Library

Catalogue”. Master’s diss., Northumbria University. http://www.preater.com/wp­content/uploads/2011/12/Andrew­Preater­MSc­Dissertation­final.pdf.

In his Master’s dissertation, Preater conducted usability testing with nine postgraduate students using cognitive walk­throughs and RGT interviews in both WebPAC, Innovative Interface’s traditional catalog, and Encore. He found that “We may tentatively conclude that Encore is a step forward as an online catalogue because there was almost complete agreement among the participants … that Encore was a better match to the ideal catalogue than was the WebPAC” (60). He also found that while many of the users liked the interface of Encore and considered it to have better relevancy ranking (despite the fact that both WebPAC and Encore used the same algorithms to determine ranking), that there were still some users who preferred the WebPAC interface to Encore, illustrating that any discovery layer cannot fully satisfy all types of uses and users.

Primo Management Group. 2010. “University of Minnesota Digital Conservancy: MNCAT Plus and MNCAT Classic Survey : Results and Analysis.” http://conservancy.umn.edu/handle/92473.

A survey undertaken at the University of Minnesota, Twin Cities in late 2009 compared graduate students’ reactions to their traditional catalog from ALEPH (MNCAT Classic) and their implementation of Primo (MNCat Plus), both Ex Libris products. At the time of the survey, MNCAT Plus was the default search for users. Survey respondents noted that they were less likely to find known items using MNCAT Plus. Interestingly, while respondents were generally favorable to the idea of including both journal articles and books, the responses indicated that either they were unaware that articles were available through MNCAT Plus, that the results from the catalog and federated search should be integrated, or that the results from the article section were not returning relevant results. The report concluded with three recommendations for discovery layers: support of multiple discovery tasks, including searching for known­items and searching by subject; enable ability for users to find items both held locally and held at other libraries; ensure that article­level data is added, easily found, and easily accessed.

Sadeh, Tamar. 2008. “User Experience in the Library: A Case Study.” New Library World 109 (1/2): 7–24. doi:10.1108/03074800810845976.

Usability testing conducted at the University of Minnesota found the interface of Primo to be “friendly, easy to use, and easy to learn.” (22). Issues that arose during testing included problems understanding terminology and the icons used; however, the author notes that these

One Search To Rule Them All: Mapping the Literature on User Experience & Discovery ACRL 2013 | Indianapolis IN | Poster Session

Annotated Bibliography 13 Greene & Moore April 5, 2013

and other issues were corrected before Primo’s general release. Thoburn, June, Annette Coates, and Graham Stone. 2012. "Simplifying Resource Discovery and

Access in Academic Libraries: Implementing and Evaluating Summon at Huddersfield and Northumbria Universities." In Planning and Implementing Resource Discovery Tools in Academic Libraries, ed. Diane Dallis and Mary Pagliero Popp, 580­597. Hershey, PA: IGI Global. doi: 10.4018/978­1­4666­1821­3.ch034.

While the focus on this article is primarily on implementing Serials Solutions Summon, the authors describe usability testing that was undertaken at both universities. Findings confirm the ability for multi­disciplinary discovery, and that easy linking to full text is imperative for a discovery tool to be effective.

Thomas, Bob, and Stefanie Buck. 2010. “OCLC’s WorldCat Local Versus III’s WebPAC.” Library Hi

Tech 28 (4): 648–671. doi:10.1108/07378831011096295.

Usability testing conducted at Western Washington University Libraries, compared the ability of users to carry out common tasks in their traditional catalog, WebPAC Pro, and WorldCat Local. Users indicated only a slight preference for WorldCat Local over the current catalog, and they found that several tasks were more easily conducted in the traditional catalog. Problems found during testing included users not understanding the scope of WorldCat Local, the intermixing of articles and books caused confusion, and known title searching. Due to these findings, Western Washington University chose to not subscribe to WorldCat Local, but to monitor whether these shortcomings had been addressed and revisit the decision then.

Thompson, JoLinda, Kathe Obrig, and Laura Abate. 2013. “Web­Scale Discovery in an Academic

Health Sciences Library: Development and Implementation of the EBSCO Discovery Service.” Medical Reference Services Quarterly 32 (1): 26–41. doi:10.1080/02763869.2013.749111.

This article presents information about implementation of a discovery tool at a health sciences library. Part of stated reason for selecting the EBSCO product was the large proportion of full­text made available within the results. Library staff focus groups elicited concerns about relevancy, about the presence of multiple subject facets, and about the lack of a language facet on the results screen. (The multiple subject facet problem was subsequently addressed by EBSCO.) Tests with users revealed the following problems: difficulty in finding advanced search and search history; users didn’t understand a small icon was for the link resolver, resulting in confusion about full text; users often failed to see all facets; sometimes ‘old’ facets carried forward. That said, participants liked facets the best of the features and the response was overwhelmingly that they would use them. The authors positively noted on the willingness of vendor to work with library to make changes and develop additional features.

Thomsett­Scott, Beth, and Patricia E. Reese. 2012. “Academic Libraries and Discovery Tools: A

Survey of the Literature.” College & Undergraduate Libraries 19 (2­4): 123–143.

One Search To Rule Them All: Mapping the Literature on User Experience & Discovery ACRL 2013 | Indianapolis IN | Poster Session

Annotated Bibliography 14 Greene & Moore April 5, 2013

This literature review provides an excellent all­in­one treatment of over sixty recent articles published on selection, implementation and user testing of various discovery layer products by academic libraries, covering all the major themes we identified in the literature overall.

Varnum, Ken. 2011. “Serials Solutions’ Summon: Familiarity Breeds Success.” Library Journal 136

(Reference Supplement) (November 15): 18.

Customers can query the Summon web service and display results within their own interfaces – the University of Michigan has taken this approach and branded their service ArticlesPlus. Varnum reports that usage of ArticlesPlus increased considerably as compared to their previous federated search tool, and that three­quarters of survey respondents indicated they were satisfied with the tool in winter 2011, although advanced users mentioned the loss of previous functionality enabling users to search specific, tailored combinations of databases.

Ward, Jennifer L., Steve Shadle, and Pam Mofield. 2008a. “Chapter 3: User Experience, Feedback,

and Testing.” Library Technology Reports 44 (6): 17–23.

The University of Washington conducted early usability tests on WorldCat Local with both undergraduate and graduates students. They found that while users were generally able to find materials, there were issues with the detailed results screen, label wording, and the placement of buttons.

Way, Doug. 2010. “The Impact of Web­scale Discovery on the Use of a Library Collection.” Serials

Review 36 (4): 214–220. doi:10.1016/j.serrev.2010.07.002.

Doug Way of Grand Valley State University Library studied Summon’s impact on usage of the collection following their fall 2009 launch of the product. At that time Summon did not provide usage statistics [though this functionality appears to have been added in early 2011]. Through analysis of data available from other sources, he determined that a “steep increase in full­text downloads and link resolver click­throughs suggest[ed] … a dramatic impact” (219). Further, he found that access of both the library catalog and of “aggregated news content” via Summon increased over time. (See also Miller, Jonathan. 2011. “Usage Statistics for Summon.” The Director’s Blog. http://rollins­olin­library.blogspot.com/2011/02/usage­statistics­for­summon.html)

Willette, Kerri, Christophe Andersen, Molly Beestrum, Shirley Bennett, Cole Robertson, and Amy

Wainwright. 2009. “Columbia College Chicago Library ­ VuFind Usability Report.” www.lib.colum.edu/CCCLibrary_VuFindReport.pdf.

Although students found VuFind easy to use, some stated that the placement of the facets on the right side of the screen (an interface particular to the I­Share implementation of VuFind) made them difficult to see: “[the students] do not look at the right hand side of the screen. ‘I

One Search To Rule Them All: Mapping the Literature on User Experience & Discovery ACRL 2013 | Indianapolis IN | Poster Session

Annotated Bibliography 15 Greene & Moore April 5, 2013

would never have seen that,’ one undergraduate said, ‘that side is always just ads” (9). Williams, Sarah, and Anita Foster. 2011. “Promise Fulfilled? An EBSCO Discovery Service Usability

Study.” Journal of Web Librarianship 5 (3): 179–198. doi:10.1080/19322909.2011.597590.

In 2011, Williams and Foster published the results of a fall 2010 usability study of the Illinois State University Milner Library’s implementation of EDS, branded “Search It.” They reported generally positive reactions from participants and noted that test subjects made use of both pre­ and post­search limiters (facets), particularly the ‘peer­reviewed’ limiter. No study participant looked at records past the first page, and users seemed disproportionately likely to select citations with images. No participants used the Integrated Search (federated search) functionality, and only one participant even noticed the feature, though users generally seemed to grasp that it contained multiple types of items and noted this positively.

One Search To Rule Them All: Mapping the Literature on User Experience & Discovery ACRL 2013 | Indianapolis IN | Poster Session