Assessment for Impact:

Turning Data into Tangible Results

Laura MillerTim MortonPaul Rittelmeyer

Library Assessment ConferenceSeattle, WA

August 5, 2008

• Balanced Scorecard

• LibQUAL+

• Surveys• Users • Staff

• Focus Groups

• Follow-up Interviews

• Statistics (circ, ILL, gate counts, purchase requests,

etc.)

Where Does Data Come From?

• Improve Library Services

• Inform Collections Decisions

• Influence Satisfaction in the

Workplace

• Support Budget Requests

• Contribute to Staff Development

What Does Data Do?

Metric U.4.b. Turnaround time for new book and ILL requests (external resources).

Target1: Satisfy the turnaround targets 75% of the time.Target2: Satisfy the turnaround targets 50% of the time.

Method: New Books: Fill requests in 7 days.  A sample of in-print US titles requested by University-affiliated patrons will be drawn from the request database by Management Information Services. Turnaround time from patron request to availability will be measured.ILL: Fill requests in 7 days. Turnaround time will be measured by ILS from patron request to notification that the book or article is available.

Improve Library ServiceRush Order (aka Purchase Express)

Improve Library Service

Acquisitions(order to receipt)

2-4 days

AUL

OK

Improve Library Service

Acquisitions(order to receipt)

2-4 days

Cataloging(receipt to shelf prep to

notifying patron)1 day

AUL

OK OK

Improve Library Service

Acquisitions(order to receipt)

2-4 days

Cataloging(receipt to shelf prep to

notifying patron)1 day

AUL

OK OK

Balanced Scorecard

Improve Library Service

Acquisitions(order to receipt)

2-4 days

Cataloging(receipt to shelf prep to

notifying patron)1 day

Patron request/selector review/placing order

?

AUL

OK OK

Balanced Scorecard

Improve Library Service

Acquisitions(order to receipt)

2-4 days

Cataloging(receipt to shelf prep to

notifying patron)1 day

Patron request/selector review/placing order

?

AUL

OK OK

Balanced Scorecard• Selector training• Back ups for email/order placement

• Funding for expedited shipping

Inform Collections DecisionsLibQUAL+ and satisfaction with journals

Inform Collections DecisionsLibQUAL+ and satisfaction with journals

LibQUAL+ 2006UVa Faculty and Graduate Student Ratings of Journal Collections

5.00

6.00

7.00

8.00

9.00

ArchitectureFaculty

EducationFaculty

EngineeringFaculty

HumanitiesFaculty

Science/MathFaculty

SocialScienceFaculty

ArchitectureGrads

EducationGrads

EngineeringGrads

HumanitiesGrads

Science/MathGrads

SocialScienceGrads

• Focused on low-scoring areas

• Approached a diverse group of faculty

• Asked for specific needs, wants and

suggestions

• Kept the interviews brief

Targeted InterviewsFollowing up with faculty

Inform Collections Decisions

• Dissatisfaction was not caused by single factor – no ‘smoking

gun’

• Perceived access was a problem – in one area we already

subscribed

to all but one of the requested journals

• Lack of foreign titles

• Missing or incomplete backfiles

What We Learned?

Inform Collections Decisions

• To address access, we instituted a major, ongoing re-design of our

website

• We invested in a electronic resource management system (Serials

Solutions 360)

• We prioritized the purchase of science backfiles (Wiley, Elsevier) and

new publisher packages

• In the arts, we put significant funds into the purchase of quality

images, or, if they could not be bought, we digitized them ourselves

The Results

Inform Collections Decisions

Influence Staff Satisfaction

Library Worklife Survey• Conducted in even-numbered years

• Available to all staff via online form with email reminders

• Response Rates: 2004 – 60%, 2006 – 52.2%

Two-Step Analysis/Response Process

1.Online survey• Concerns identified

2.Focus groups, one-on-one meetings• What can be done to address concerns?

Influence Staff Satisfaction

Problems Identified Solutions Implemented

• Staff don’t know about other departments

• Open Houses for departments

• Staff don’t know what’s going on system-wide

• “Did You Know?”, Job/Position Announcements

• Administration not available to hear concerns

• Brown Bag Lunches, Open Door Policies

Impacts Seen 2006 2004 Change

• Library staff in other departments are familiar with what I do. 28.3% 17.3%

11.0%

• The Library administration is willing and available to listen to 52.3% 37.3%

15.0% my concerns, comments, and recommendations.

• The Library administration effectively communicates to staff. 36.9% 36.6%

0.3%

• Collaboration is encouraged and supported in my job. 69.0% 75.8% -

6.8%

• I get cooperation from other departments when we work together. 71.2% 74.0% -

2.8%

Influence Staff Satisfaction

Problems Identified Solutions Implemented

• Need more staff and better distribution of work

• 24 new positions created

• Inadequate facilities, technology and support resources

• Library IT involved with allocations• Facilities staff increased• Balanced scorecard metric for facilities improvement

Impacts Seen 2006 2004 Change

• I have adequate resources and materials to complete assignments. 59.5% 49.2%

10.3%

• I feel safe in my work environment. 87.9% 81.5% 6.4%

• The Library is concerned about and addresses my ergonomic needs 67.0% 76.3% -

9.3%

• I am physically comfortable in my work environment (temperature, light, noise). 56.4% 44.6% 11.8%

Influence Staff Satisfaction

Problems Identified Solutions Implemented

• Like Administration, HR not always available

• New HR staff hired• HR moves from Admin area into library

• Salaries are inadequate • Extra allocations towards salary adjustments

• No opportunity for growth • Better advertising of existing policies• Mentoring programImpacts Seen 2006 2004 Change

• My salary and benefits are reasonable for the work I do. 43.9% 21.5% 22.4%

• The salary I receive is equitable when compared to co-workers in similar grade levels or ranks. 49.0% 30.9% 18.1%

• My work at the Library helps me achieve my career goals. 55.4% 35.9% 19.5%

• There are opportunities for me to advance at the Library. 36.8% 28.1% 8.7%

• There are adequate staff development opportunities provided by the Library. 64.2% 57.3% 6.9%

Influence Staff Satisfaction

The Overall Impact

2006 2004Change

• My work at the Library helps me achieve my career goals. 55.4% 35.9%

19.5%

• I am satisfied with my job. 63.2% 58.5%

4.7%

• Overall Job Satisfaction Category 65.5% 58.0% 7.5%

Support Budget Requests

• 80% of Music faculty responded to the 2004 Faculty Survey. Of these respondents, 88% chose sound recordings as their highest priority, making it the highest priority item for the department.

• While every Music faculty respondent rated the resource “sound recordings” on a 1-5 scale, the category received one of the lower ratings for satisfaction, a 3.88.

• The rating for sound recordings among faculty as a whole was 3.89.

Support Budget RequestsMusic Library Budget

• Raising awareness of traditional practices as assessment tools (ref counts, gate counts, circ data, ILL data)

• Need to review, revise, reassess our BSC metrics and other tools. Assessment is ongoing.

• Managers and all staff need to learn how to better use data and assessment results in daily operations.

• Continue to move assessment out of MIS only. Position MIS as consultant/coordinator or assessment done throughout the organization.

• Assessment is for everyone!

Challenges AheadPromoting a Culture of Assessment