Project web site – www.arl.org/libqual/

TM

Ottawa, CanadaJune 14, 2006

Presented by:

Dr. Colleen Cook, Dean

Texas A&M University

Quantitative and Qualitative Methods for Assessingand Improving Outcomes and Service Quality

Why Assess?Why Assess?

““In an In an age of accountabilityage of accountability, there is a pressing , there is a pressing need for an effective…process to evaluate and need for an effective…process to evaluate and compare research libraries.” compare research libraries.”

700 participants in LibQUAL+700 participants in LibQUAL+™™

123 Association of Research Libraries (ARL) 123 Association of Research Libraries (ARL) alone, over alone, over $3.4 billion dollars$3.4 billion dollars were expended in were expended in 2003/20042003/2004

Note. M. Kyrillidou and M. Young. (2005).ARL Statistics 2003-04. Washington, D.C.: ARL, p.5.

Libraries Remain a CredibleLibraries Remain a CredibleResource in 21Resource in 21stst Century Century

Note. Digital Library Federation and Council on Library and InformationResources. (2002). Dimensions and Use of the Scholarly Information Environment.

98% agree with statement98% agree with statement, “My … library , “My … library contains information from credible and contains information from credible and known sources.”known sources.”

Changing BehaviorsChanging Behaviors

Note. Digital Library Federation and Council on Library and InformationResources. (2002). Dimensions and Use of the Scholarly Information Environment.

Recent Survey:Recent Survey:Only Only 15.7% agreed with the statement15.7% agreed with the statement “The “The Internet has not changed the way I use the Internet has not changed the way I use the library.”library.”

Faculty: Dependence onElectronic Resources Will Increase

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2000 2003

Not WellSomewhatVery Well

“I will become increasingly dependent on electronic research resources in the future.”

http://www.arl.org/arl/proceedings/144/guthrie_files/guthrie.ppt

Research Behavior:Research Behavior:Personal ControlPersonal Control

When searching for When searching for printprint journals for journals for research:research:

Note. Digital Library Federation and Council on Library and InformationResources. (2002). Dimensions and Use of the Scholarly Information Environment.

• Only 13.9% ask a librarian for assistanceOnly 13.9% ask a librarian for assistance• Only 3.2% consider consulting a librarian a preferred Only 3.2% consider consulting a librarian a preferred way of identifying informationway of identifying information

Total Circulation

Note. M. Kyrillidou and M. Young. (2005). ARL Statistics 2003-04. Washington, D.C.: ARL, p.6.

Reference Transactions

Note. M. Kyrillidou and M. Young. (2005). ARL Statistics 2003-04. Washington, D.C.: ARL, p.6.

Web UsageWeb UsageTotal File Requests - UT Austin Libraries 2000-2003Total File Requests - UT Austin Libraries 2000-2003

0

100,000,000

200,000,000

300,000,000

400,000,000

500,000,000

600,000,000

700,000,000

800,000,000

900,000,000

2000 2001 2002 2003

Total Hits

Enter LibQUAL+™

The necessity of assessment

Rapid shifts in information-seeking behavior

The reallocation of resources from traditional services and functions

The Challenge ofAssessment in Libraries Traditional statistics emphasize inputs, expenditures,

acquisitions, holdings, etc. Helping funding agencies understand success of

investment No demonstrable relationship between expenditures and

service quality Lack of metrics describing outcomes: success from the

user’s point of view Need to redesign library services to better meet

changing patterns of use Building the climate, tools, and skill set for library

assessment

ARL New MeasuresInitiative Collaboration among member leaders with

strong interest in this area Specific projects developed with different models

for exploration Intent to make resulting tools and methodologies

available to full membership and wider community

LibQUAL+™ Goals

Improve mechanisms and protocols for evaluating libraries

Develop web-based tools for assessing library service quality

Identify best practices in providing library service Support libraries seeking to understand changes

in user behavior Assist libraries seeking to re-position library

services in the new environment

LibQUAL+™ Outcomes

Securing information that contributes meaningfully to planning and improvement efforts at a local level

Providing analytical frameworks that institutional staff can apply without extensive training or assistance

Helping decision-makers understand success of investments

Finding useful inter-institutional comparisons

PERCEPTIONS SERVICE

“….only customers judge quality;

all other judgments are essentially

irrelevant”

Note. Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press.

The LibQUAL+™ Premise

13 LibrariesEnglish LibQUAL+™ Version

4000 Respondents

QUAL

QUAN

QUAL

QUAL

QUAN

QUAL

PURPOSE DATA ANALYSIS PRODUCT/RESULTDescribe library environment;build theory of library service quality from user perspective

Test LibQUAL+™ instrument

Refine theoryof service quality

Refine LibQUAL+™ instrument

Test LibQUAL+™ instrument

Refine theory

Unstructured interviewsat 8 ARL institutions

Web-delivered survey

Unstructured interviews at Health Sciences and the Smithsonian libraries

E-mail to surveyadministrators

Web-delivered survey

Focus groups

Content analysis:(cards & Atlas TI)

Reliability/validityanalyses: CronbachsAlpha, factor analysis,SEM, descriptive statistics

Content analysis

Content analysis

Reliability/validity analyses including Cronbachs Alpha,factor analysis, SEM, descriptive statistics

Content analysis

VignetteRe-tooling

Iterative

Emergent2000

2005700 Libraries English, Dutch, Swedish,

German LibQUAL+™ Versions160,000 anticipated respondents

LibQUAL+LibQUAL+™ Project™ Project

Case studies1

Valid LibQUAL+™ protocol

Scalable process

Enhanced understanding of user-centered views of service quality in the library environment2

Cultural perspective3

Refined survey delivery process and theory of service quality4

Refined LibQUAL+™ instrument5

Local contextual understanding of LibQUAL+™ survey responses6

76 Interviews Conducted

York University University of Arizona Arizona State University of Connecticut University of Houston University of Kansas

University of Minnesota University of Pennsylvania University of Washington Smithsonian Northwestern Medical

LoadedPT:P1:01xxxxxxxxxxxxxx.txt,S:\Admin\Colleen\ServQual Interviews\TEXT Only\01xxxxxxxxx.txt (redirected: c:\zz\atlasti\fred

Dimensions ofLibrary Service Quality

Empathy

InformationControl

Responsiveness

Symbol

Utilitarian space

Assurance

Scope of Content

Ease of Navigation

Self-Reliance

Library as Place

LibraryServiceQuality

Model 3

Refuge

Affect of Service

Reliability

Convenience

Timeliness

Equipment

Affect of Service

“I want to be treated with respect. I want you to be courteous, to look like you know what you are doing and enjoy what you are doing. … Don’t get into personal conversations when I am at the desk.”

Faculty member

Library as Place

“One of the cherished rituals is going up the steps and through the gorgeous doors of the library and heading up to the fifth floor to my study. … I have my books and I have six million volumes downstairs that are readily available to me in an open stack library.”

Faculty member

Library as Place

“I guess you’d call them satisfiers. As long as they are not negatives, they won’t be much of a factor. If they are negatives, they are a big factor.”

Faculty member

Information Control

“…first of all, I would turn to the best search engines that are out there. That’s not a person so much as an entity. In this sense, librarians are search engines [ just ] with a different interface.”

Faculty member

Information Control

“By habit, I usually try to be self-sufficient. And I’ve found that I am actually fairly proficient. I usually find what I’m looking for eventually. So I personally tend to ask a librarian only as a last resort.”

Graduate student

Multiple Methodsof Listening to Customers Transactional surveys* Mystery shopping New, declining, and lost-customer surveys Focus group interviews Customer advisory panels Service reviews Customer complaint, comment, and inquiry capture Total market surveys* Employee field reporting Employee surveys Service operating data capture*A SERVQUAL-type instrument is most suitable for these methodsNote. A. Parasuraman. The SERVQUAL Model: Its Evolution And Current Status. (2000).

Paper presented at ARL Symposium on Measuring Service Quality, Washington, D.C.

LibQUAL+™ Resources

An ARL/Texas A&M University joint developmental effort based on SERVQUAL.

LibQUAL+™ initially supported by a 3-year grant from the U.S. Department of Education’s Fund for the Improvement of Post-Secondary Education (FIPSE)

Initial project established a expert team, re-grounded SERVQUAL concepts, and designed survey methodology

Survey conducted at over 500 libraries resulting in a data base of over half a million user responses

NSF funded project to refocus LibQUAL+™ on the National Science Digital Library (NSDL)

Participating Libraries

World LibQUAL+™ Survey 2005

Rapid Growth in Other Areas

Languages American English British English French Dutch Swedish

In development Chinese Greek Spanish German

Consortia Each may create 5 local

questions to add to their survey

Types of Institutions Academic Health Sciences Academic Law Academic Military College or University Community College European Business Hospital Public State

Countries U.S., U.K., Canada, the

Netherlands, South Africa, Sweden, France, Australia, New Zealand, Malaysia

“22 items”

2000 2001 2002 200341-items 56-items 25-items 22-items

Affect of Service Affect of Service Service Affect Service Affect

Reliability Library as Place Library as Place Library as Place

Library as Place Reliability Personal ControlInformation Control

Provision of Physical Collections

Self-RelianceInformation Access

Access to Information

Access to Information

Survey Instrument

“And a Box”

Why the Box is so Important About 40% of participants provide open-

ended comments, and these are linked to demographics and quantitative data.

Users elaborate the details of their concerns.

Users feel the need to be constructive in their criticisms, and offer specific suggestions for action.

Reliabilityalpha By Language

By LanguageService Info. Lib as

Group n Affect Control Place TOTALAmerican (all) 59,318 .95 .91 .88 .96British (all) 6,773 .93 .87 .81 .94French (all) 172 .95 .90 .89 .95

Reliability alpha by University Type

By University TypeService Info. Lib as

Group n Affect Control Place TOTALComm Colleges 4,189 .96 .92 .89 .974 yr Not ARL 36,430 .95 .91 .88 .964 yr, ARL 14,080 .95 .90 .87 .96Acad Health 3,263 .95 .92 .90 .96 

Validity Correlations

Validity Correlations Serv_Aff Info_Con LibPlace TOTALperServ_Aff 1.0000 .7113 .5913 .9061 Info_Con .7113 1.0000 .6495 .9029 LibPlace .5913 .6495 1.0000 .8053 TOTALper .9061 .9029 .8053 1.0000ESAT_TOT .7286 .6761 .5521 .7587 EOUT_TOT .5315 .6155 .4917 .6250  

Understanding Understanding LibQUAL+™ ResultsResults

• Measures the distance between minimally Measures the distance between minimally acceptableacceptable and and desireddesired service quality ratings service quality ratings

• Perception ratings ideally fall within the Perception ratings ideally fall within the Zone Zone of Toleranceof Tolerance

Key to Bar Charts

LibQUAL+™ 2004 Summary Collegesor Universities American English

(n = 69,449)

Score Norms

Norm Conversion Tables facilitate the interpretation of observed scores using norms created for a large and representative sample.

LibQUAL+™ norms have been created at both the individual and institutional level

Institutional Norms for PerceivedMeans on 25 Core Questions

Note: Thompson, B. LibQUAL+ Spring 2002 Selected Norms, (2002).

LibQUAL+™ InteractiveInstitution Statistics

YEAR (REQ.) – defaults to current year of the survey; controls the Master List (dynamic)

Master List of Institutions

All

Clear

• OVERALL – defaults to OVERALL

• Dimension (3)

mindesperadeq sup

SUBMIT

VARIABLES - Default to perceived

Results Page: This page generates a graph, Summary Statistics, Your Statistics, and Norms (including users’ norm values); restates information entered into the form

4 items included in this

drop-down menu

Your List

INSTITUTION TYPE

SAVED LISTS

CONSORTIA

LANGUAGE ADD

ADD

ADD

ADDLanguage (s) will be determined based on

selection (s) from the Master List of Institutions; dynamic

User selects an institution from the Master List, the selection is then added to “Your List”. To avoid duplicate choices, the selection from the Master List will disappear once added to “Your List”.

Other parameters can be added below (institution type, language, consortia, and/or SAVED LISTS).

Consortia, based on

current year; dynamic

1)

2)

SAVEText here stating that this section is optional and may be added to “Your List” to narrow down results.

Text box for user to name and save search parameters

for future searches.

LibQUAL+™

Adequacy GapThe difference between the minimum and perceived score

In Closing LibQUAL+™

Focuses on success from the users point of view (outcomes)

Demonstrates that a web-based survey can handle large numbers; users are willing to fill it out; and survey can be executed quickly with minimal expense

Requires limited local survey expertise and resources Analysis available at local and inter-institutional levels Offers many opportunities for using demographics to

discern user behaviors

LibQUAL+™ Resources

LibQUAL+™ Website:http://www.libqual.org

Publications: http://www.libqual.org/publications

Events and Training: http://www.libqual.org/events

LibQUAL+™ Bibliography: http://www.coe.tamu.edu/~bthompson/servqbib

LibQUAL+™ Procedures Manual: http://www.libqual.org/Information/Manual/index.cfm

LibQUAL+™ Contact Information MaShana Davis

Technical Communications Liaison mashana@arl.org

Richard Groves Customer Relations Coordinator richard@arl.org

Mary Jackson LibQUAL+™ Services Manager richard@arl.org

Martha Kyrillidou Director, ARL Statistics and Service Quality Programs martha@arl.org

woofwoof