Search to Discovery: Finding Global Scholarly

Resources with Primo

Pascal Calarco &

Alison Hitchens, LibraryDecember 6, 2011

Agenda

• The state of search in libraries (Pascal)• Expanding Primo beyond the local

catalogue (Alison)• Questions

2011

Library Information Systems: Milestones

Discovery

Metasearch

Citation Linking

ILS 3rd gen (Client-server; 1990s)

ILS 2nd gen (Mainframe; 1980s)

OCLC (library network; 1972)

Early systems

MARC

1960 1980 19902000 2010

2011

In the beginning, there was the card catalog (1901+)

Indexes:• Subject• Author• Title• Interfiled cards, call

number access

2011

Library of Congress National Union Catalog (pre-1956)

2011

Henriette Avram, Developer of MARC

• Programmer/analyst at Library

Of Congress• Developed system for printing

card catalog information (MARC)• ISO certification 1973

2011

Later, there was the Online Public Access Catalog (OPAC)

• Machine Readable Cataloging (MARC)• Inventory of the print/physical holdings of a

library• Better than the card catalog; keyword searching

& boolean functionality• Non-intuitive; required training or intermediation

(information professional)• Limited generally to single library

2011

Library networks & resource sharing

2011

Print to Electronic

2011

Now: Electronic Almost Ubiquitous

• 85%+ of journal literature digital• Hundreds of specialized scholarly

databases• Mass print book digitization efforts• Electronic books going mainstream• Aggregated meta-indexes: 750 million

metadata for journal/newspaper articles

2011

Goal: improve user experience

• Users want to FIND not search• Source required information to user

regardless of format or location• Leverage our knowledge of academic

community @ uWaterloo• Integrate into key services: LMS, CMS,

other library services

2011

Database Content SilosContent Silos

System Silos

Catalog ILL Meta-search

eReserveWebsite

Science-Direct

Web of Science

ETDs EEBOJSTOR

Metasearch: an interim step

• aka Federated Search; emerged 2003 • Distributed search from one interface via

web services, SOAP/XML gateways• Idiosyncratic and slow; vendors

implemented variously• Relevancy of merged results problematic

2011

Problems with catalog searching & evolution to discovery

• UCLA & Berkeley: information retrieval & user behavior (1986-1996)

• Google Books: “digitize the world’s knowledge” (2002)

• Karen Schneider, Andrew Pace, Roy Tennant: “The OPAC ‘Sucks’”(2002)

• Next generation catalogs -> Discovery (2008+)

2011

Catalogs: Information Science Research

• Christine L. Borgman (1986) “Why are online catalogs hard to use? Lessons learned from information retrieval studies” Journal of the American Society for Information Science

• Ray R. Larsen (1991) “The decline of subject searching: Long-term trends and patterns of index use in an online catalog” Journal of the American Society for Information Science

• Ray R. Larsen (1992) “Evaluation of advanced retrieval techniques in an experimental online catalog” Journal of the American Society for Information Science

• Ray R. Larsen (1996) “Cheshire II: designing a next-generation online catalog” Journal of the American Society for Information Science

• Christine L. Borgman (1996) “Why are online catalogs still hard to use?” Journal of the American Society for Information Science

How Users Search: What We’ve Learned

• Most people make typos at least some of the time

• Most searches are 2, 3, 4 words with no Boolean operators

• Most searches use keyword• Search is hesitant, iterative, often random

process of discovery• Most people start elsewhere• Few read help screens• Few use advanced search – this is true even in

Google

The Google Effect

• Expectations for web search tools now:– Radically simplified UI, fast results– Aggregated content – Relevant results on first page– Natural Language queries– Spelling correction/adaptation

2011

The OPAC “Sucks”• The OPAC lacks common features of most search

engines– Relevance ranking vs. last in, first out

– Spell checking (related - did you mean?)

– Popular query operators like + and –

– Refine search

– Sort flexibility

– Faceting

– Citation indexing vs full text

– Developed for print materials, limitations with electronic materials or atomized items (like articles)

– Difficult for certain known item search

Industry Trends

• Decouple the front end (search and discovery) from the back end (inventory and cataloguing)

• Service Oriented Architecture – many programs loosely coupled

• Cloud services -- SaaS• The 5th generation of library business

systems emerging now – hosted, cloud solutions

Discovery Characteristics

• Enhanced Search Functionality– Faceted browse– Relevance ranking– “Did you mean?” / Spell Checking

• auto-correction, resubmit search– Content aggregation

• Integrating search for books, articles, etc.– Single, Simple Search Box– FRBR – functional requirements for bibliographic

record, grouping editions

Discovery Characteristics, cont.

• Enhanced Experience– Sometimes fun and engaging– Interactive/Collaborative– User centered design

• Enhanced Services– Find it / Get it for me– Book Covers / Synopsis– Full text– Availability on same page as results

Discovery Characteristics, cont.

• Enhanced Content– Article Searching– Commercial Data– Merging Special Collections– Harvesting Online Collections

• Grey Literature• Free Content

• Enhanced Access– Syndication - Getting into users tools

• Course Management Systems• Browser and Desktop Tool Bars

• Portals

Discovery Components1. Next Generation Catalog

2. Next Generation “Unified Search” Aid

Normalization &Apache

SOLR/Lucene

User Interface

ILS

OPAC

MARC

VendorData

MetaSearch

OAIVendorData

Circ Data

FullText

Primo Central

Content Components

Primo

RACER

TUG

Archives

OCUL

Geospatial

HathiTrust

Others

Phase I Phase II Future

Evolution of Discovery

Catalog

Primo

Meta-search

Primo

Central

Options for Expanding Primo

• Local ingestion of resources using FTP or OAI harvesting

• Searching remote resources in Primo using the Primo DeepSearch API*

• Subscribing to a large centralized index, such as Primo Central

*Application Programming Interface

2011

Local ingestion of records

• Example: Hathi Trust Digital Library– Harvest the public domain records from Hathi

Trust Digital Library– Normalize the records– Index the records in our local Primo database– Schedule updates from Hathi Trust into Primo

2011

Normalization: creating local sort field (Date – Oldest)

2011

Primo Normalized XML (PNX)

2011

Open source & Open platform

• Primo uses Lucene for its indexing• SOLR exposes Lucene as a web service

and allows for faceting• APIs and web services allow flexibility and

customization

2011

We can’t index everything!

• Trying out a subscription to Primo Central, a centralized index of scholarly journal articles, newspapers, conference proceedings etc.

• User sees one interface; user is searching 2 indexes

2011

What is Primo Central Index?• A centralized index

– of free and restricted resources– primarily articles & e-books– based on metadata & full-text provided by

publishers/aggregators– based on the collections selected by the

library in the Primo Administration module– created & maintained by our vendor, Ex Libris

What is Primo Central Index?

• A centralized index– of records harvested using the same process

as our local Primo database– created using the same PNX record structure

as our local Primo database– indexed using the same indexing tools as our

local Primo database

Blending local and remote resources

• Both local and remote results are represented in the facets

• Blended relevance ranking– Can configure Primo to boost high ranking local

results so that when Primo is doing relevance ranking on our 4 million records alongside 100s of millions of Primo Central records local results aren’t missed by the user

Search = local resources & Primo Central

How does it work?

• Ex Libris has created & indexed records for millions of items based on information from the publishers

• Primo searches Primo Central the same way it searches the local database

• Full text availability is determined in advance by our URL resolver SFX, i.e.

• Delivery of the resource uses menu for

New features: snippets give context

If your search term is found in the full-text, Primo supplies a snippet highlighting the term

New features: expanding the search

Defaults to our library’s electronic subscriptions but users can expand the search to all of Primo Central

New Facets & Facet Values

Added value: bX Recommender

Trouble-shooting remote resources

• We can view the PNX records using web services but we have no control over the content or the normalization rules

• Records have the same structure as our local records but are missing local fields and don’t reflect local policies

2011

Assessing Primo Central

• Over 65 hours of one-on-one usability testing and focus groups with undergraduate students, graduate students, faculty, staff and alumni

• Library staff survey• Feedback form• Statistics from Cognos

2011

Looking to the future• What other content should be added to

Primo?• How can we improve/enhance the

interface?• What is the right balance for boosting local

physical resources?• How do we point users to resources that

can’t be searched using Primo?

2011

Questions?

• Pascal Calarco– Associate University Librarian, Digital &

Discovery Services– pvcalarco@uwaterloo.ca

• Alison Hitchens– Cataloguing & Metadata Librarian– ahitchen@uwaterloo.ca

2011