gain spring meeting, mercer university, macon, ga april 24, 2014 measuring your impact: using...

G A I N S P R I N G M E E T I N G , M E R C E R U N I V E R S I T Y , M A C O N , G AA P R I L 2 4 , 2 0 1 4

MEASURING YOUR IMPACT: USING EVALUATION FOR LIBRARY ADVOCACY

2

INSTRUCTOR

Andrew YoungkinNN/LM SEA Evaluation and Technology Coordinator

ayoungki@hshsl.umaryland.edu

3

AGENDA

Morning

9:00 Introductions and Overview of Workshop

9:20 Demonstrating the Library’s Value – Mission & Vision

9:50 The Environment (SWOT exercise)

10:10 The Plan - Logic Model

11:00 Break/Group exercise

11:30 The Plan – EvaluationCreate your own, share

Afternoon

12:00

LUNCH

1:00 Data, Tools, and Analysis

2:00 Communicating the Value

3:00Create your message – elevator speech, infographic, annual report introduction

3:45 Resources, class questions and evaluation

4

WORKSHOP OBJECTIVES

By the end of the workshop you will:• Understand your library’s value in terms of the

mission of the larger organization• Describe some tools used to assess your library,

its users and stakeholders• Identify the tools and methods used for data

collection and analysis• Design an evaluation plan for a service in your

library• Communicate evaluation results

5

WHAT THIS WORKSHOP IS NOT…

• Custom-tailored• A quick fix• One size fits all

6

“WHAT GETS MEASURED, GETS DONE.“

Proving to your administration that money spent in your library is going to pay student achievement dividends is well worth the time and effort

http://www.doug-johnson.com/dougwri/what-gets-measured-gets-done.html

7

DEMONSTRATING YOUR LIBRARY’S VALUE

8

WHY DEMONSTRATE VALUE?

• To show the impact of your library on the larger organization’s mission and goals

• To show accountability for your resources; your library’s contribution to the bottom line

• To demonstrate that your library provides value-added services

• To use as an advocacy and marketing tool

9

HOW & WHAT TO EVALUATE

• What you decide to evaluate depends on:• What you need or want to know• What your users feel is important• What certain stakeholders want to have evaluated

• Use “smallball” evaluation• Base hits or home runs?

10

LIBRARY SERVICE IS VALUE-ADDED

• Cost savings• Institutional & Personal

• Time savings• Travel, discovery, skills acquisition, opportunity for other

uses of time

• Access• Opportunity to use/apply information

• Impact on learning, research

11

THE BOTTOM LINE

• You are either generating revenue, or supporting those who do

OR …

• You are helping to control operating expenses or supporting those who do

OR …

• You are creating expenses that add recognized value

OR …

• You are creating expenses that must be controlled or eliminated to reduce overhead

Will Welton, PHD, Director, MHA Program, University of Washington

12

THE BIG QUESTION!

What is the bottom line and what is your library doing to support it?

13

WHAT YOU NEED IS A PLAN

• Know the vision• Look at the environment• Create the plan• Collect and analyze the data• Communicate the value

14

THE VISION

15

KNOW WHAT YOU WANT TO DO AND WHY

• Align your library’s mission & goals with the organization’s mission & goals

• Your library’s value = library’s contribution to achieving organization’s mission & goals

16

MISSION STATEMENTS

• dedicated to cultivating the inherent gifts, skills and talents of children in Grades 6 through 12 so that they are prepared for success in college and life.

• to provide resources that support the curriculum, help our students to develop habits to become lifelong learners and self-actualized citizens, and promote understanding and appreciation of our Marianist heritage.

17

MISSION STATEMENTS

• ...foster in our students academic, physical and creative fulfillment, together with strength of character, while helping them become productive members of our school community.

• …support and … foster the active and creative pursuit of academic and artistic achievement through the effective use of information and the encouragement of curiosity and scholarly inquiry.

18

HAIKU

creative fulfillment,academic and physical;productive community

...foster in our students academic, physical and creative fulfillment, together with strength of character, while helping them become productive members of our school community

effective information useencourages curiosity andscholarly achievement

…support and…foster the active and creative pursuit of academic and artistic achievement through the effective use of information and the encouragement of curiosity and scholarly inquiry

19

WRITE A HAIKU

• Write your mission statement or another message• Turn that into a haiku• Standard haiku is 5/7/5 syllables, but English

haikus are more flexible because of our syllable structure

• Share your haiku

20

MADE TO STICK

• SUCCESs• Simple – core, compact• Unexpected – gets attention• Concrete – specific, not abstract • Credible – testable• Emotional – more than self interest• Stories – stimulate, inspire

(From: Made to Stick. Heath, Chip. New York: Random House, 2007)

THE ENVIRONMENT

21

22

WHY LOOK AT THE ENVIRONMENT?

• Understand needs, desires and problems in context

• Validate assumptions about your contributions and services

• Provide a baseline for future evaluation• Help to develop the blueprint to plan and

evaluate your contributions and services

23

COMPONENTS OF ENVIRONMENT

• The organization• Your library• Clients/users• Stakeholders• The community

24

ASSESSING THE ENVIRONMENT

• User/stakeholder input• Surveys, focus groups, interviews• Unsolicited feedback• Conversations

• Observation• Assessment by walking around• Know the language of your users

• Information/literature• Journal/serial/magazine

25

ASSESSING THE ENVIRONMENT

• Library statistics and records• Collection use• Librarians’ work including reference, teaching, etc.

• Get out of your library and talk to people• Understand the role of information in their work• Learn how information is acquired, applied• Discover information problems they are trying to solve

• SWOT analysis• Understanding strengths, weaknesses, opportunities,

threats• Consider how these affect your work, your assessment

26

LIBRARIANS’ TOP 10 TASKS

• School Librarians say…1. Help students select books2. Collaborate with teachers3. Read and book talk with classes4. Teach research and use of technology to students and teachers5. Keep library organized including cataloging, weeding,

inventory, late notices, vacuuming, dusting6. Study standards to prepare library lessons7. Troubleshoot technology issues around the building8. Help with extracurricular activities including laminating, school

plays, lunch duty, etc9. Prepare book, equipment and supply orders10. Organize special library events like book fairs, author visits,

contests

27

LIBRARIANS’ TOP 10 TASKS

• School Principals say…1. Help students access information and books2. Help faculty access information and books3. Share technology expertise with students and teachers4. Select “appropriate” materials5. Model love for reading6. Collaborate with teachers7. Provide equipment and technology8. Provide leadership with technology9. Teach research skills, teach about books and databases10. Provide an inviting environment

How does your boss see you? Proof that principals value librarians. School Library Journalhttp://www.slj.com/2012/09/careers/how-does-your-boss-see-you-proof-that-principals-value-librarians/#_

28

• Librarians say

• Principles say

29

SWOT ANALYSIS

Weaknesses Internal

◦ Statements about what is lacking in your library

Opportunities External

◦ What do stakeholders want you to do that you are not doing?

Threats External

◦ Factors that can adversely impact your library’s goals

Strengths Internal

◦ Positive statements about your library

30

PLAN BACKWARD, IMPLEMENT FORWARD

31

THE LOGIC MODEL - A TOOL TO…

• Help organize your thoughts – both before and during a project/program

• Provide a framework for planning and evaluating programs

• Clarify intended outcomes

• Acts as a communication tool

Good source for more information:W.K. Kellogg Foundation Logic Model Development

Guide (pdf on your thumb drive; see class bibliography for

link)

32

PIECES OF THE LOGIC MODEL

• Outcomes• Activities• Resource• Outputs

33

OUTCOMES

• Outcomes - changes in attitude, behavior, skills, knowledge or situation• Short, medium and/or long term• Intended or unintended• Positive as well as negative• S.M.A.R.T.

34

SMART OUTCOMES

Outcome: Health resources Web pages are available on the library website by the end of the project

• Specific: one or more web pages with links to EBM resources are available on the library website

• Measurable: done or not; do nurses access resources

• Action-oriented: created and posted; nurses access resources

• Realistic: It is possible within the timeframe

• Timed: “when” is articulated

35

OUTCOMES

Outcome• Long Term:

• Patients and their families will have improved health information literacy

• Intermediate Term:• Staff will have information for timely clinical decision-making

• Short Term:• Health resources Web pages are available on the library

website by the end of the project

36

ACTIVITIES AND RESOURCES

• Activities • What will you do?

• Resources• What you have

• Income• Equipment• Collection

• What you need• Operating expenses (e.g., personnel, acquisitions,

maintenance, etc.)• Funds for new initiatives or services

37

OUTPUTS

• Outputs• How many did you do?• How many attended?• How many were distributed?• How many times was it used?

38

Resources

Activities/Outputs

Outcomes

If we get theseresources…

Conduct theseactivities and deliver these products

We will see these outcomes

PUTTING IT ALL TOGETHER

39

THE MODEL MAY CHANGE OVER TIME

Activities

OutputsOutcomes

Resources

40

LOGIC MODEL

Resources Activity Outputs Outcomes

41

LOGIC MODEL

Resources Activity Outputs ST/IT/LTPersonnel, money, expertise needed

What you will do What your activity will produce – data, classes, brochures, etc.

The SO WHAT & WHEN– the benefits that accrue as a result of your program

Knowledgeable librarian Select web resources List of URLs and descriptions of resources/sites

HTLM expertise Create web page(s) Write HTML code or edit a template/existing page(s)

Health resources Web pages are available on the library website by the end of the project

Tracking software Collect page hit statistics; survey staff about usefulness, timeliness of web based information

Web statistics; survey results

Staff will have information for timely clinical decision-making

Outcomes: ST: Health resources Web pages are available on the library website by the end of the project; IT: Staff will have information for timely clinical decision-making

42

LOGIC MODEL GROUP ACTIVITY

• Identify outcome(s)• Articulate short/intermediate/long term• List

• activities • outputs• resources

43

THE EVALUATION PLAN

44

THE EVALUATION PLAN

• Builds on the logic model• Evaluates success• Assesses value

45

CONSIDER THE PURPOSE

• Who is your audience? This may be different than the beneficiaries of your services• Your users• Administrators• Patients/families

• How will the information be used?• Financial savings or justification• Intangible or non-monetary value of program

benefits to community• Marketing and advocating for your library

46

WHAT DO YOU WANT TO KNOW?

Have you achieved your stated outcome?• Short Term:

• Health resources Web pages are available on the library website by the end of the project

• Intermediate Term:• Staff will have information for timely clinical decision-making

• Long Term:• Patients and their families will have improved health

information literacy

47

WHEN DO YOU WANT TO KNOW?

• Consider objective(s)• Articulate the time frame

48

EVALUATION PLAN

Activity Indicator

Target Data Source

Data Analysis Methods

Data Collection Frequency

What is done to achieve outcome – from Logic Model

Measurable result of activity

Specific indicator measurement desired

Origin of indicator measurements

Organize, examine, learn from the data

Date, time and intervals

Outcome (from logic model):Staff will have information for timely clinical decision-making

49

EXAMPLE PART 1

Activity Indicator Target

What is done to achieve the outcome

Definition of success

Specific desired result

Create web pages; analyze use statistics; survey potential users

Web pages are available; Users indicate the information was available for clinical decision making

50% of respondents report information was available for timely [define timely here] clinical decision- making

Outcome (from logic model):Staff will have information for timely clinical decision-making

50

EXAMPLE PART 2

Data Source Data Collection Frequency

Data Analysis Methods

Origin of indicator measurements

Date, time and intervals

Organize, examine, learn from the data

Web statistics, survey

Data recorded quarterly throughout fiscal year; twice annual survey

Spreadsheets; survey coding

51

NOW DESIGN YOUR OWN EVALUATION PLAN

52

FROM YOUR LOGIC MODEL…

• Select an outcome and describe:• An activity that will help achieve the outcome• Indicators of outcome results• Targets for indicators• Data source• Data collection frequency• Data analysis methods

53

MAKING SENSE OF THE DATA

54

ANALYSIS

• Think about data analysis before collecting the data

• Identify experts that can help with data analysis• Test your collection and analysis tools• Leave enough time• Go “smallball” if needed

55

YOUR DATA

• Quantifiable• Statistics

• What do you collect?• What do you do with it?

• Evidence of change• Measurable impact of services provided

• Qualitative• Stories, opinions• Evidence of change• Perceived impact of services provided

56

SOME TOOLS

• Benchmarking• How you compare to others of similar size and

circumstance• A common measuring stick to evaluate process

performance • Qualitative/quantitative analysis

• Coding narratives• Scoring responses

• Cost/Benefit Analysis• Use to evaluate the benefits of a program or service

• Return on Investment• How much your investment in the program or

project earned for the company

57

BENCHMARKING

• Can improve your library's performance• Can help you gain upper management support• Can help prove the value of your library

58

BENCHMARKING RESOURCES

• Your own library

• The competition

• Other libraries in your system, industry

• Associations:• ARL – LibQual+• AAHSL – Annual survey• NN/LM MAR library value study

59

QUALITATIVE AND QUANTITATIVE

• Narrative• Code text for concepts• Determine frequency of concepts• Describe/illustrate results

• Quantitative• Assign values to possible responses• Calculate mean, median, mode of response values

60

CBA: COST/BENEFIT ANALYSIS

• The ratio showing dollar value of benefits gained for dollar value of costs

• Use to evaluate the benefits of a program or service

• Benefits divided by costs gives the value realized by transaction

61

CBA FORMULA

BENEFITS ÷ COSTS

Or

BENEFITSCOSTS

62

ROI: RETURN ON INVESTMENT

• Percentage showing the return (increase in value) on dollars spent to achieve a benefit• Amount the company’s investment in a program earned

for the company• Earnings on money spent

63

ROI FORMULA

((BENEFITS – COSTS) ÷ COSTS) × 100

Or

BENEFITS – COSTS COSTS

× 100

64

PREPARING FOR CBA AND/OR ROI

• Select the program or service to value • Journal subscriptions; searching the literature class

• Identify benefits derived from program/service • Users have access to literature for decision making

• Identify who gets the benefit • Library users, patients, care team

• Convert the benefits to quantifiable terms (based on surveys and/or records)

• # of journals used; # of articles accessed; time available for research, patient care

65

EXAMPLES OF BENEFITS

• Books used (in house or borrowed) • Print journals used (in house or borrowed) • E-journal articles accessed• Patron time saved• Value of information available for research,

teaching, patient care

66

COSTS

• Purchase price• Staff time• Space, light, heat, computers, benefits

67

COSTING TIPS

• Consider tangible vs. intangible benefits

• Be conservative in estimating benefits and liberal in assessing costs

• Obtain administrative approval of the value of the benefits for each alternative; involve financial staff

• Express benefits and costs in a value system shared by all parties – $$

• Think about inflation and depreciation if study extends over several years – only if the CFO insists!

68

A CBA/ROI EXAMPLE

Subscription to online Fictional Medical Journal

at Anytown Institution

69

JOURNAL SUBSCRIPTION

• The library spends $1,600 for an institutional subscription to Fictional Medical Journal (FMJ) online and $200 to manage the subscription

• 7,800 articles are viewed per year using the library’s subscription

70

ASSUMPTIONS

• Each person reads on average 2 articles per week• 2x52=104 articles per user

• 7,800 uses represents 75 users• 7,800/104=75

• Without a library, each person who needed FMJ would subscribe at a cost of $99 for online only

71

COSTS AND BENEFITS

• Cost: $1,800• $1,600 for subscription and $200 for staff time to

process/maintain

• Benefit: $7,425• 75 personal subscriptions: 75 x $99

72

CBA MATH

BENEFITSCOSTS

$7,425 = $4.125

$1,800

$4.13 benefit to the institution for each dollar spent by the library on FMJ

73

ROI MATH

$7,425 – $1,800

$1,800

= 312.5%

Net Costs

Benefits – Net Costs× 100

× 100

74

REPLACING YOUR LIBRARY OR A LIBRARY SERVICE

• What is the value of the resources and services your library provides?

• What would your users pay on the open market?• Would your institution pay the retail costs?• Would your institution pay for everyone who

needs information or for select staff?

Retail value calculatorhttp://nnlm.gov/mcr/evaluation/calculator.html

75

CALCULATORS THAT DO THE MATH FOR YOU

• Retail value calculator• http://nnlm.gov/mcr/evaluation/calculator.html

• CBA/ROI calculator• CBA: Benefits/Costs• ROI: ((Benefits-Costs)/Costs) *100

• http://nnlm.gov/mcr/evaluation/roi.html

76

77

CALCULATOR DATA POINTS

• Where do the values come from?• What does each line mean?• http://nnlm.gov/mcr/evaluation/valuation.html

78

Becker Library

79

80

81

CBA/ROI EXERCISE

• Select one service or resource• List all the costs associated with providing the

service once or obtaining one resource• List alternative sources for that service or

resource• List the costs associated with that alternative

source• Plug the numbers into the formula

82

COMMUNICATING THE RESULTS

83

USE THE RESULTS…

• To communicate your value• To promote and advocate for your library• To maintain or get additional funding• As baseline information to show improvement

over time• To improve library services and programs

84

SOME COMMUNICATION METHODS

• Written and/or oral reports• Formal • Informal

• Graphics

• Executive summary

• Press release for newsletter or local paper

• Promotional materials for your library

• Staff meetings

• Informal hallway or elevator talk

85

MADE TO STICK (REDUX)

• SUCCESs• Simple – core, compact• Unexpected – gets attention• Concrete – specific, not abstract • Credible – testable• Emotional – more than self interest• Stories – stimulate, inspire

(From: Made to Stick. Heath, Chip. New York: Random House, 2007)

86

87

88

89

90

91

TIPS FOR WRITING

• Frame message in terms of added value to the mission and goals of the organization

• Summarize in one paragraph then support the message with evidence.

• Use figures, tables, graphs to make your point• Should be able to stand alone• Have a narrative to give more details

• Have others review the communications

92

TIPS FOR TALKING

• Know and rehearse your message

• Talk to administrators one-on-one when possible

• Focus on benefits

• Don’t use library jargon

• Educate, but don’t bore

93

“ELEVATOR TALK”

• Simple, memorable mini-speech

• Ready for any brief encounter

• What your library contributes and why that matters

• Create curiosity

Gardner, Reed S. Making the Case for Your Library: A How-To-Do-It Manual. How-To-Do-It Manuals for Librarians, Number 104. New York: Neal-Schuman, 2001: 6

“Sadly and too often, the people who have the most influence over our future are really not all that interested in libraries.”

94

ELEVATOR TALK EXERCISE

• Select an outcome • Invent results• Determine the audience• Create

• a sound bite that highlights the value of the outcome to that audience

OR• an interesting but mysterious statement that will make

your listener want to know more

95

GO FORTH & EVALUATE

• Do smallball evaluation• Relate your value to the mission and goals of your

organization • Plan your evaluation before you begin• Share your value with many audiences• Don’t wait for a crisis – start now!