library assessment

LIBRARY ASSESSMENT

Jennifer Rutner, Senior Analyst, Ithaka S+R Pratt Institute

Human Information Behavior September 22, 2011

Senior Analyst, Ithaka S+R

School:

BA in Religious Studies, 2002

MLIS from Pratt, 2005

Enrolled in Quantitative Methods for the Social Sciences at Columbia, 2010-?

Work:

Assessment & Planning Librarian, Columbia University Libraries, 2006-2011

Chair, ACRL Assessment Committee, 2011

ASSESSMENT LIBRARIANSHIP

Assessment

“To assess, in general, is to determine the importance, size, or value of; to evaluate. Library staff assess operations by collecting, interpreting, and using data to make decisions and improve customer service.”

ARL Spec Kit #303, Library Assessment, December 2007

What we talk about when we talk about “assessment”

assessment Evaluation

User research

Quality assurance

ROI/Value

Impact

Culture of Assessment

A Culture of Assessment is an organizational environment in which decisions are based on facts, research and analysis, and where services are planned and delivered in ways that maximize positive outcomes and impacts for library clients.

A Culture of Assessment exists in organizations

where staff care to know what results they produce and how those results relate to customer expectations.

Amos Lakos: www.usc.edu/.../locations/leavey/news/conference/presentations/presentations_9-16/Assessment/UCLA_Lakos.ppt

Why assess?

Why assess?

Stuff I know Stuff I know I don't know

Stuff I don't know I don't

know

How is data used in libraries?

• Strategic planning + management

• Decision making

• Program evaluation

• Advocacy

• Budgeting

• Regular service improvements

Assessment Mission (CUL)

“Serve library users and staff through the gathering, analysis, and application of high-quality, actionable information to guide library decision making.”

The Research Process

Establish research

questions

Environmental scan

Gather available

information

Information Needs

(unknowns)*

Establish priorities

Assign methodology

Develop tools Test tools

IRB Conduct

assessment Analysis

Decision making

The Assessment Process

Brainstorming Information Need

Priority Methodology Formal Question

Outcome

“I want to know…”

RESEARCH METHODS

Surveys

Use random sampling to generalize to the broader population.

Set questions, with no opportunity for follow-up.

Allows for statistical analysis.

Questionnaires Evaluation tool: How was it?

Exploratory tool: What do you do?

Preferences tool: What do you want?

Reporting tool: What did you do?

Analysis stops at descriptive statistics.

Response Rate vs. Representativeness

“It ain’t response rate.”

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

% of population

% of respondents

Focus Groups “Researchers attempt to capture peoples explanations”

• Answering “how?” “why?” and “what?” questions – open ended question

• Look at a topic up-close, rather than get a panoramic view of the entire issue

Focus groups can uncover insights and perspectives that are not retrievable by other methods, from a small group of participants.

Focus Groups

Standard

– Uses a rigid set of questions

Guided

– Uses a set of topics to explore, but the question wording is flexible

Exploratory

– The most informal, questions arise through the course of the conversation about the topic

Ethnography

Methodologies

– Work study

– Photo essays

– Dream-catcher workshops

– Mapping diaries

– Pilot projects

– Interviews

Observation

Wait, watch, write. Head Count+

• Sections of the library

• Type of seating

• Group vs. individual

• Food/drink

• Technology

Way Finding: 3D Usability

How do users navigate space?

– Provide task

– Record steps to completion

– Record completion rate

– Record challenges

Learning Outcomes

• Rubrics

• Pre- and post-tests

• Minute Papers

– What’s one useful thing you learned today?

– What’s one thing you’re still confused about?

How does this impact their academic success?

What are they learning?

What are we teaching?

Usability Studies

Methods

• Card sort

• Heuristiv evaluation

• Paper prototyping

• Personas

• Task analysis

• Work-study

Evaluates

• Ease of use

• Efficiency of use

• Memorability

• Error frequency and severity

• Subjective satisfaction www.usability.gov

ROI

“For every dollar invested in the library, the college/university/school/community/business received X dollars in return.”

“For every dollar invested in the library, the library produces X services, which can be valued at Y.”

Institutional Review Board (IRB)

• Protect human subjects.

• Ensure research is ethical.

• Ensure research complies with federal and state laws.

Protocols include:

• Research questions and hypothesis

• Subject profiles

• Study procedures

• Recruitment materials

• Report on findings

LIBQUAL+

LibQual+ Survey 2009 www.libqual.org

“22 questions and a box” Affect of Service

Information Control

Library as Place

Comments?

http://www.libqual.org/

LibQual+ Items

Affect of Service AS-1 Employees who instill confidence in users AS-2 Giving users individual attention AS-3 Employees who are consistently courteous AS-4 Readiness to respond to users' questions AS-5 Employees who have the knowledge to answer user

questions AS-6 Employees who deal with users in a caring fashion AS-7 Employees who understand the needs of their users AS-8 Willingness to help users AS-9 Dependability in handling users' service problems Library as Place LP-1 Library space that inspires study and learning LP-2 Quiet space for individual activities LP-3 A comfortable and inviting location LP-4 A getaway for study, learning, or research LP-5 Community space for group learning and group

study

Information Control IC-1 Making electronic resources accessible from my

home or office IC-2 A library Web site enabling me to locate information

on my own IC-3 The printed library materials I need for my work IC-4 The electronic information resources I need IC-5 Modern equipment that lets me easily access

needed information IC-6 Easy-to-use access tools that allow me to find things

on my own IC-7 Making information easily accessible for

independent use IC-8 Print and/or electronic journal collections I require

for my work Local Questions Providing help when and where I need it Making me aware of library services Availability of subject assistance Ability to navigate library Web pages Access to archives, special collections

Response: Representativeness

Response by status across the University matches the population distribution very closely.

Greatest difference: 8%

This is representative data!

http://www.libqual.org/documents/admin/Representativeness.pdf

Status % of responses

% of population

Undergraduates 40.03% 32.38%

Graduates 53.21% 55.21%

Faculty 6.78% 12.40%

Response: Representativeness

Response by discipline across the University matches the population distribution nearly perfectly.

E.g. We’re not missing anyone!

http://www.libqual.org/documents/admin/Representativeness.pdf

Reading LibQual+ Charts

Desired

Minimum

Perceived/Reality Zone of Tolerance

Superiority Gap

Adequacy Gap

Undergraduate Students

4

5

6

7

8

9

AS-1 AS-2 AS-3 AS-4 AS-5 AS-6 AS-7 AS-8 AS-9 IC-1 IC-2 IC-3 IC-4 IC-5 IC-6 IC-7 IC-8 LP-1 LP-2 LP-3 LP-4 LP-5

Graduate Students

4

5

6

7

8

9


Faculty Responses

4

5

6

7

8

9


Library Staff

4

5

6

7

8

9


FACULTY PERCEPTION OF JOURNAL COLLECTIONS

Following up…

The Infamous IC-8

“The print and/or electronic journal collections I require for my work.”

ARL Libraries 2009

4

5

6

7

8

9

10

AS - 1 AS - 2 AS - 3 AS - 4 AS - 5 AS - 6 AS - 7 AS - 8 AS - 9 IC-1 IC-2 IC-3 IC-4 IC-5 IC-6 IC-7 IC-8 LP - 1 LP - 2 LP- 3 LP - 4 LP - 5

Figure 3: LibQUAL+ 2009, ARL Libraries Faculty

ARL Libraries 2009: IC8

Have perceptions changed?

4

5

6

7

8

9

2006 2007 2008 2009

Figure 5: LibQUAL+ 2006-09, Faculty Ratings of Journal Collections, ARL Libraries

M=8.58, SD=0.10

M=7.44, SD=0.17

M=6.77, SD=0.40

M=8.61, SD=0.07

M=7.52, SD=0.15

M=7.01, SD=0.25

M=8.57, SD=0.13

M=7.51, SD=0.21

M=7.15, SD=0.39

M=8.56, SD=0.10

M=7.52, SD=0.14

M=6.96, SD=0.35

Is there a difference in scores from year to year? (ANOVA)

• 2006-2009 adequacy gaps from each ARL institution.

• P-value = 0.119, which is not deemed statistically significant.

Faculty were no more or less dissatisfied with journal collections in 2009.

Journals and Overall Satisfaction

5

6

7

8

9

-1.2 -1 -0.8 -0.6 -0.4 -0.2 0

Figure 8: LibQUAL+ 2009, Correlation of Faculty Satisfaction with Journal Collections (IC-8) and Overall Library Service, 21 Libraries

r =0.71

IC Item 2006 2007 2008 2009 Standard

Deviation

(by question)

Mean

(by

question)

IC-1 0.44 0.61 0.44 0.55 0.08 0.51

IC-2 0.71 0.55 0.61 0.42 0.12 0.57

IC-3 0.86 0.42 0.31 0.73 0.26 0.58

IC-4 0.71 0.67 0.4 0.61 0.14 0.60

IC-5 0.72 0.49 0.27 0.45 0.19 0.48

IC-6 0.73 0.58 0.42 0.12 0.26 0.46

IC-7 0.81 0.46 0.40 0.67 0.19 0.59

IC-8 0.80 0.60 0.55 0.71 0.11 0.67

What else should we be watching?

-0.7

-0.6

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

IC1 IC2 IC3 IC4 IC5 IC6 IC7 IC8 2006 (n=37 )

2007 (n=19)

2008 (n=14)

2009 (n=21)

Figure 6: LibQUAL+ 2006-09, Information Control Adequacy Gaps Over Time

What do our faculty say?

Support

“What would be great for faculty would be if when things are not available, there was one source in the library, extraordinarily skilled at tracking down items. […] These people would be specialists in working the electronic and journal capabilities.”


Search and Online Access

“I think just having free text search, like Google book search, would be something that would be very, very useful to have. I still feel like we are living 20 years behind where the rest of the world is in terms of being able to search these databases and large collections of books that we have.”


Work-Arounds

“I just buy them individually from my research funds, so it’s coming out of my research money.”


Quick List

“If I was to give a suggestion, maybe to have discipline-specific pointers that could help each discipline find things. […] It’s more of an interface issue than a collections issue.”


Resources

“The size of the collection is not as important as getting the current collection working as smooth as possible. Before, when we used to go to the library, we got service.”


Print vs. Electronic

“A few years ago, I wouldn’t have said that. But, I guess things have changed.”

2CUL HUMANITIES PHD STUDENTS

The 2CUL Project

“Columbia and Cornell University Libraries are pleased to join forces in a transformative and enduring partnership between our two great library systems that enables us to pool resources to provide content, expertise, and services that are impossible to accomplish acting alone.”

http://2cul.org/

Research Procedures

Spring 2010 • Ethnographic Training • Focus groups (5 total) Summer + Fall 2010 • Interviews (45 total)

– 90 minutes, individual

• Post-questionnaire (paper) Winter 2010-11 • Analysis and reporting

$

870 pages of transcripts.

The Student

Personal Space

Previous Experience

Personal Expectations

Self-determination

Personal Life

The Institution + Department

Institutional Space

Funding

Dept. Requirements + Expectations

Culture + Community

Advising

Teaching

Attrition

Research + Writing

Coursework

Exams + Preparation

Prospectus + Preparation

Discovery

Writing Process + Revision

Defense + Preparation

Information Management

The Library

Librarians

Collections

Services

Library Space

Technology

The Profession

Job Search

Publishing

Challenges

Successes

Opportunities

First

Second

Third

Code Level

Fourth

Code Tree

Time from BA graduation Through Expected PhD Completion

Graduation from undergrad to start of PhD Start of PhD to candidacy Candidacy to PhD expected graduation

14

14

10

21

24

3

24

22

8

19

16

6

2

2

2

3

2

8

3

2

1

5 1

2

5

3

2

3

1

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Grad Program

Funding

Advising (Columbia only)

Library Services

Library Collections

Library Spaces (Columbia only)

Humanities PhD Student Study Questionaire Please rate your overall satisfaction with the following at CU:

Very Satisfied

Satisfied

Neutral

Dissatisfied

Very Dissatisfied

n/a

0

1

2

3

4

5

6

7

8

9

10

1< 1 -2 hrs 2 - 4 hrs 4 - 6 hrs 6 - 8 hrs 8+

No

. of

Stu

de

nts

Time Spent in the Library by Frequency and Duration

Daily Weekly Monthly

0% 20% 40% 60% 80% 100%

browse

write

read

office hours

research w. library …

research w. non-library …

computers

consult librarian

meet colleagues

other Post-Exam Pre-Exam

Activities in the Library by Status

Percentage of Students

30

10

5

Humanities PhD Student Study Questionnaire Have you visited any non-CU libraries to use their collections

for dissertation research?

Yes No n/a

39

2

4

Humanities PhD Student Study Questionnaire Do you receive financial support from CU for the academic

year?

Yes No n/a

6

34

5

Humanities PhD Student Study Questionnaire Do you have an outside job that provides income?

Yes No n/a

Findings: Provide Space

"The thing that has been the best for me is having a space to work. I got more done last year when I had my locked carrel than I had gotten done in years before or since, because it was a dedicated space in which I could keep all of my sources [...]."

Findings: Foster Community

“It’s having community. Belonging to your community, having friends that are doing this and feeling that you have something worthwhile to say that other people are recognizing it.”

Findings: Provide Access to Deep Research Collections

“I have to say that I have had every resource that I have needed from the library. I really can’t say, ‘Here I am in the sixth year because you didn’t buy that set of resources for me and I don’t have the materials to work with, so how can you expect me produce work?’”

Findings: Provide Research, Information Management, and Teaching Expertise Assistance

“[…] maybe sitting down with an advanced research reference librarian . . . might be in my best interest as I go into the writing stage of my paper, just so that I can make sure I am not saying something that has already been said or duplicating research, or that I am not missing something that is cutting-edge and that's really important to my argument.”

Findings: Developing Scholarly Identity

“I had to tell my committee in an email, I plan on having a draft of the first chapter to you by June. If I don't, please get on my case… So, I actually found that I needed to make deadlines for myself and then tell them so that they knew, and even though they wouldn't care, my pride was at stake at that point.”

ITHAKA S+R

ITHAKA is a not-for-profit organization that helps the academic community use digital technologies to preserve the scholarly record and to advance research and teaching in sustainable ways.

We pursue this mission by providing innovative services that aid in the adoption of these technologies and that create lasting impact. .

Mission

• Ithaka S+R is a strategic consulting and research service that focuses on the transformation of scholarship and teaching in an online environment, with the goal of identifying the critical issues facing our community and acting as a catalyst for change.

• JSTOR is a research platform that enables discovery, access, and preservation of scholarly content.

• Portico is a digital preservation service for e-journals, e-books, and other scholarly e-content.

Our Services

Ithaka S+R Surveys

2009 Faculty Survey

3,025 Faculty responses

8.6% of population

Focus on research institutions Conducted 2000, 2003, 2006,

2009, 2012

2010 Library Director Survey

239 Library Directors responded

13% of population

9 Carnegie Classification Levels:

– 79 responses are doctoral

– 66 are master’s

– 94 are baccalaureate

Ithaka S+R Staff: Roger Schonfeld, Ross Housewright, Matt Long

0% 20% 40% 60% 80% 100%

Gateway

Archive

Buyer

Research Supporter

Undergraduate Information Literacy Teacher

Teaching Facilitator

How important to you is it that your college or university library provide each of the functions below? (Percentage answering very important.)

Faculty Members Library Directors

The Role of the Library: Comparison with Faculty

Note: Faculty member data are from Ithaka S+R Faculty Survey 2009.

Digital vs. Print Spending

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Five years from Now

Now

Journals Books All other items

“What percentage of your library’s materials budget is spent on the following items?”

Digital vs. Print Spending

Journals Books

Directors predict a

106% drop in spending on print journals in the

next 5 years…

…bringing budget shares to:

12% Print

88% Digital / 54%

Print

46% Digital /

Directors predict a

46% drop in spending on print books in the

next 5 years.

…bringing budget shares to:

Print to Electronic Transition: Publishing

“I am completely comfortable with journals (that I use regularly/ that my library subscribes to) ceasing their print versions and publishing in electronic-only form.”

Faculty Members Library Directors

Agree 70% Neither agree

nor disagree 22% Disagree 8%

39% Agree

30% Neither agree nor disagree

32% Disagree

Print to Electronic Transition: Existing Collections

“Within the next five years, the use of (online or digitized journals/ electronic versions of scholarly monographs) will be so prevalent among faculty and students that it will not be necessary to maintain library collections of hard-copy…

… Books.” … Journals.”

Agree 54% Disagree 13%

7% Agree

59% Disagree

Library Spending Priorities

If you received a 10% increase in your library's budget next year in addition to the funds you already expect to receive, in which of the following areas would you allocate the money? (Please check up to three areas in the following list that you would invest in.)

0% 20% 40% 60% 80% 100%

Staff in management/administration of …

Electronic versions of scholarly …

Other digital resources

Facilities expansions and renovations

Staff for reference and user services/ …

Tools for discovery (OPACs, indices, …

Online or digital journals

Library Spending Priorities

If you received a 10% increase in your library's budget next year in addition to the funds you already expect to receive, in which of the following areas would you allocate the

money?

0% 20% 40% 60% 80% 100%

Developing and maintaining special …

Supporting the research projects of faculty …

Building or maintaining local discovery resources

Purchasing/ licensing digital resources

Providing reference services

Supporting faculty instruction and student …

Percentage ranking this item as 1

Percentage ranking this item as 2

Library Staffing Priorities

Ideally, how would you prioritize your staff resources in the following areas? Please rank the items by order of importance.

Discovery: A Declining Role for the Library?

0%

10%

20%

30%

40%

50%

60%

70%

80%

2006 "Now" "5 years from now"

Library Directors

Faculty Members

Percentage answering that it is very important that the library serve as a "gateway"

User Needs Assessment

94% Informal discussions with faculty and students or emails soliciting feedback

71% Locally developed polls or surveys

49% Focus groups or test sessions

37% Cross-institutional polls or survey (such as Libqual+)

16% Structured Interviews

10% Ethnographic studies

8% With the help of outside consultants

6% Other

In the past 2 years, has your library regularly solicited feedback about services or collections from library users in any of the following ways? (Please check all that apply.)

Only 13% do not have a formal means to assess user needs.

35% of respondents agreed that “My

library has a well-developed strategy to meet changing user needs and research habits.”

Library Strategies

CHALLENGES

challenges • Lack tradition of using data for improvement

• No assessment advocate within organization

• Library staff lack research methodology abilities

• Weak analysis and presentation of data

• Inability to identify actionable data

• Library “culture” is skeptical of data

• Leadership does not view as priority/provide resources

• Library organizational structure is “silo-based”

• Staff do not have sufficient time

Turning Results into Action: Using Assessment Information to Improve Library Performance, Steve Hiller (University of Washington) , Stephanie Wright (University of Washington)

http://libraryassessment.org/bm~doc/hiller.pps

challenges

THANK YOU [email protected]

References About Assessment

• “Value of Academic Libraries: A Comprehensive Review and Report,” Megan Oakleaf: http://www.ala.org/ala/mgrps/divs/acrl/issues/value/val_report.pdf

• Focus Groups: A Practical Guide for Applied Research by Richard Krueger and Mary Anne Casey

• ARL SPEC Kit #303 on Library Assessment, December 2007

• Keys to Effective, Sustainable, and Practical Assessment Steve Hiller, Martha Kyrillidou, and Jim Self http://www.arl.org/arldocs/stats/statsevents/laconf/2006/HillerSelf.ppt

• www.libraryassessment.info (blog)

• Columbia Assessment Program: https://www1.columbia.edu/sec/cu/libraries/bts/assessment/index.html

Interesting Studies

• Studying Students, The Undergraduate Research Project at the University of Rochester http://www.ala.org/ala/mgrps/divs/acrl/publications/digital/Foster-Gibbons_cmpd.pdf

• For fun: http://www.coolinfographics.com/

• Cornell University Libraries ROI: http://research.library.cornell.edu/value

• Ellsevier/UIUC ROI: http://www.slidefinder.net/t/library_strategic_investment/illinois_roi_study/1313459

• University of Arizona), Learning in an Online Environment: Assessment of an Online Information Literacy Credit

• Course, Yvonne Mery, Jill Newby, Ke Peng: http://libraryassessment.org/bm~doc/2010_lac_poster.pdf

• University of Chicago: Wayfinding Revisited: Improved Techniques for Solving Usability Problems in Physical Spaces Agnes Tatarka, David K. Larsen

• LibQual+ Survey Literature: www.libqual.org/Publications

• 2CUL Humanities PhD Study:

Conferences

• Library Assessment Conference Proceedings: http://libraryassessment.org

• Northumbria Conference Proceedings: http://www.lib.sun.ac.za/Northumbria7/Programme.htm

Ithaka S+R

• http://www.ithaka.org/ithaka-s-r/research/faculty-surveys-2000-2009

• http://www.ithaka.org/ithaka-s-r/research/ithaka-s-r-library-survey-2010

http://www.ala.org/ala/mgrps/divs/acrl/issues/value/val_report.pdf

http://www.arl.org/arldocs/stats/statsevents/laconf/2006/HillerSelf.ppt

http://www.libraryassessment.info/

https://www1.columbia.edu/sec/cu/libraries/bts/assessment/index.html

http://www.ala.org/ala/mgrps/divs/acrl/publications/digital/Foster-Gibbons_cmpd.pdf



http://www.coolinfographics.com/

http://research.library.cornell.edu/value

http://www.slidefinder.net/t/library_strategic_investment/illinois_roi_study/1313459

http://libraryassessment.org/bm~doc/2010_lac_poster.pdf

http://libraryassessment.org/bm~doc/tatarka.pps



http://www.libqual.org/Publications

http://libraryassessment.org/

http://www.lib.sun.ac.za/Northumbria7/Programme.htm

http://www.ithaka.org/ithaka-s-r/research/faculty-surveys-2000-2009












http://www.ithaka.org/ithaka-s-r/research/ithaka-s-r-library-survey-2010













