the aahsl/arl partnership in exploring outcomes assessment through libqual+

The AAHSL/ARL Partnership

in Exploring Outcomes

Assessment through LibQUAL+

Surfing the Tsunami of Service Quality

2

Tamera Lee, Director, Greenblatt Library, Medical College of Georgia, Chair, AAHSL Task Force on Service Quality Assessment

James Shedlock, Director, Galter Health Sciences Library, Northwestern University, Editor of AAHSL Annual Statistics, Task Force Liaison

Rick Forsman, Director, Denison Memorial Library, University of Colorado Health Sciences Center,

AAHSL President 2002/Board Liaison


3

What is LibQUAL +

A new measurement initiative of the Association of Research Libraries and researchers at TAMU (Colleen Cook, Fred Heath and Bruce Thompson)

Define quality to complement the traditional quantitative library measures (number of volumes, amount of space, size of staff, etc.)


4

What is LibQUAL+?

National, broad-scale project to define and measure service quality across institutions and identify best practices

Web-based survey based on research grounded in gap analysis from the service quality business literature Parasuraman, Zeithaml and Berry


5

What is LibQUAL+?

Specifically the survey indicates how user expectations compare with user perceptions of service quality

Measure gaps between what service dimensions users most desire, what they expect and perceive they are getting


6

Purpose/Task Force Outline the qualitative efforts of the

AAHSL Task Force on Service Quality Assessment:

Jane Blumenthal-Georgetown University

Diana Cunningham-NY Medical College

Rick Forsman-University of Colorado

Tamera Lee-Medical College of Georgia

Logan Ludwig -Loyola University-Chicago

James Shedlock-Northwestern University

Julie Sollenberger-University of Rochester

Susan Starr-University of California, San Diego

Laurie Thompson-State University of New York, Upstate


7

The Association of Academic Health Sciences

Libraries (AAHSL)

One of two consortia’s in the 2002 pilot of

LibQUAL+™; the other was the Ohio

Library and Information Network (OhioLINK),

a consortium of Ohio’s college and university

libraries and the State Library of Ohio.

Focus: the AAHSL/ARL partnership, the AAHSL

aggregate experience, results and collaborative

outcomes.


8

Background

With a long history of compiling and analyzing comparative statistics to serve as useful benchmarks for member libraries, AAHSL recognizes the increasing importance of outcomes measures to be used as performance indicators.


9

Background

AAHSL partnered with ARL for the participation of a cross section of 36 AAHSL libraries in the spring 2002 pilot.

The AAHSL project was funded generously by the National Library of Medicine, with shared costs by AAHSL and participating libraries to support deliverables.


10

Background

The Task Force on Quality Assessment was created in July 2001 to lead AAHSL’s involvement in a service quality assessment process and evaluate LibQUAL+™ as a qualitative measurement tool for academic health sciences libraries.


11

Task Force Goals

Testing the usefulness of the instrument for benchmarking

Testing the feasibility of customizing the instrument for the health sciences

Determining the value of cross data analysis with AAHSL statistics

Gaining experience with an aggregate outcomes measurement process


12

The Process

Coordinated the cohort registration and established a listserv to facilitate efficient communication

Drafted a procedural manual to serve as a guide and created a survey administration tip sheet for participants


13

Task Force Challenges

Clarified the importance of a clean set of collective data for AAHSL and defined demographics for respondents


14


Developed 5 unique AAHSL questions in collaboration with participants and in consultation with the National Library of Medicine.

The main objective was to add service performance issues for the health sciences, especially priorities not included or covered sufficiently by the LibQUAL+™ survey.


15

Unique AAHSL Queries

Providing health information when and where I need it

Employees teaching me how to access or manage information

Environment that facilitates group study and problem solving

Access to information resources that support patient care

Having comprehensive electronic resources


16

Library Service Quality Survey

LibQual+When it comes to… My Minimum

Service Level Is

low high

My Desired Service Level Is

low high

Perceived ServicePerformance Is

low high

N/A

1) Willingness to help users 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 N/A

2) Space that facilitates quiet study

1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 N/A

3) Complete runs of journal titles

1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 N/A


17


Guided discussion of human subjects clearance and disparities of IRBs regarding issues of anonymity, age, etc.


18

The Respondents

The TF recommended that the AAHSL participants survey their entire user populations. Only those who completed the survey in its entirety and with measurable consistency were included.

Over half (50.97%) of the people who entered the survey completed it. With each of the 4 pages viewed the percentage dropped 8-20%, most significantly at the 2nd (demographics) and 3rd pages.


19

Results

Chart—percentage of AAHSL respondents by user group (N=13,976)

AAHSL - Respondents by User Group

Staff22.60%

Faculty35.50%

Grad. Stud.34.30%

Undergrad. Stud.4.60%

Library Employees3.10%


20

Results

Chart—percentage of respondents for disciplines reflecting 5% or more (N=13,976)

AAHSL-Respondents by Discipline

Hospital/Health Syst.11.83%

School of Public Health5.72%

School of Medicine53.68%

Graduate School5.00%School of Nursing

6.70%


21

Results

Chart—average scores for AAHSL aggregate rating general satisfaction from 1-9 (with 1 being “strongly disagree” and 9 representing “strongly agree”)

AAHSL - General Satisfaction

7.47 7.05 7.35

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

Satisfaction withTreatment

Satisfaction withSupport

Overall Quality ofService


22

Frequency of Library Use—AAHSL Aggregate

Most respondents reported using the Library weekly, both on the premises (35.93%) and electronically (34.84%).

Daily electronic use was reported by 31.83% and monthly use on premises was reported by 31.42%.


23

Service Dimension Gaps that fell outside “zone of

tolerance”

AAHSL aggregate—electronic resources accessible from home or office

AAHSL students—convenient business hours

AAHSL faculty—complete runs of journal titles, electronic resources accessible from home or office, comprehensive electronic resources


24

Dimensions Most Desired by AAHSL

Aggregate

Comprehensive electronic resources (8.44)

Electronic resources accessible from home or office (8.35)


25

Dimensions Most Desired by AAHSL

AggregateStudents: (undergrad and grad):

having comprehensive electronic resources (8.33-8.50), followed closely by convenient business hours (8.32-8.43)

Faculty: having comprehensive electronic resources (8.49), followed closely by making electronic resources accessible from home or office (8.46)


26

Service Affect Dimensions Most Desired

by StaffEmployees who are courteous consistently (8.55),

Willingness to help users (8.52)

Employees who have the knowledge to answer

users’ questions (8.46)

Readiness to respond to users’ questions (8.46)


27

The Results

ARL reliability results indicate higher AAHSL user expectations and levels of satisfaction compared to the ARL environment.

Responses from the general academic campus reflect a higher response rate from undergraduate students, whose expectations and needs differ from the user profile of AAHSL institutions, consisting primarily of graduate level and above.


28

The Results

Useful view of the data is through

T-scores and norm tables.

T-scores enables examination of individual score norms for faculty and students and institutional scores in relation to scores of peer institutions.


29

The Results

The overall norm table for institutions of all types shows that one health sciences library clearly stands out as exceeding expectations--even desires.

Mercer University School of Medicine Library and LRC has the only positive average superiority score, indicating overall perceptions are higher than desired.


30

Comments

Frequent strong themes shared included

1) need for more journals, especially online

2) frustration with remote access problems

3) positive comments about staff


31

AAHSL Queries“Comprehensive

Electronic Resources”

Reflected the highest desired measure by all user groups

Represented a significant gap in minimal expectations for faculty


32

AAHSL Queries“Comprehensive

Electronic Resources”The ARL research team advised the task

force to reconsider this question as an unreasonable service demand. Seemingly the sense is that comprehensive print resources may be more achievable.

For the 2003 LibQUAL+™ cohort iteration, this question was revised to provide more focus, “Electronic resources matching my information needs.”


33

AAHSL Queries

Summative data for other four AAHSL queries indicates needs for slight improvements for all user groups.


34

Participating Libraries-Outcomes

Data used to complement other assessment measures and create additional ones, such as focus groups.

A correlative test to consider the integration of qualitative and quantitative measures conducted by Lee, Dennison and Joubert showed no correlation between service affect gap score and annual expenditures.


35

Participating Libraries-Outcomes

Many shared presentations with respective senior administrations, academic senates, student government associations, physical facilities planners and library staff.

Some made immediate changes and others want to repeat the survey in 2003 for a better view. A few incorporated LibQUAL+™ data into their annual performance metrics and other review processes.


36

Task Force Recommendations for

AAHSL:

Participate in the final phase of pilot

(2003 survey administration) for a cross

section of 23 new and repeat participating

libraries to optimize our assessment and

maximize our deliverables


37

Task Force Recommendations for

AAHSLConsider if and how a qualitative data

component (such as LibQUAL+ survey results) may be used in conjunction with the annual quantitative stats for member libraries

Continue to support LibQUAL+ assessment and development, other qualitative measurement activities and future initiatives in outcomes assessment


38

Benefits-ARL Assisted in furthering the applicability

of LibQUAL+™ as a qualitative assessment tool across different types of institutions.

The special cohort assisted ARL with promotion, training and increased participation.


39

Benefits-AAHSL

Partnership created a heightened awareness of the viability of outcomes assessment and a cost efficient protocol for listening to users.

The additional five questions allowed AAHSL to include a construct for issues perceived as unique or more

significant to health sciences libraries.

Prompted further interest in testing the instrument's feasibility for a new cohort of hospital libraries.

Advanced a culture of assessment within AAHSL and for its members.

the aahsl/arl partnership in exploring outcomes assessment through libqual+

Documents

tsunami of service qualitywhat

measure service quality

aahsl libraries

university libraries

service dimensions users

aahsl task force

aahsl project

aahsl president