defining new metrics for libraries success ischool, university of toronto symposium april 23, 2014...
TRANSCRIPT
Defining New Metrics For Libraries SuccessiSchool, University of Toronto Symposium
April 23, 2014
Dr. Bill Irwin, Huron University College
Paul St-Pierre, PHD Candidate, FIMS, UWO
In
Insights: Organizational cultures in public libraries: Gatekeepers to effective outcome evaluation?
Project’s Goal
Organizational Culture & Evaluation2
The key goal of our research is to examine existing evaluation systems of public libraries, in an effort to replace the current reliance on a predominantly output based model of performance measure, with new models of evaluation based on capturing outcomes.
Project’s Goal
Organizational Culture & Evaluation3
Outcome models assist policy makers, both inside and outside of organizations, to address fundamental questions of resource allocation, delivery methods, and agency design and purpose.
Defining Outcome Evaluation
Organizational Culture & Evaluation4
Project Objectives
Organizational Culture & Evaluation5
exploring the limitations and challenges inherent in the current qualitative evaluation system
creating a better understanding of the cultural impacts and policy implications of a sociological performance framework, on library organizations, their constituencies, and their stakeholders in the establishment
capture changes in the Practice-Program-Policy continuum and the downstream implications for how libraries are viewed and valued upon the introduction of a new system of performance measurement
OPLA Child and Youth Services Committee Teen Services Benchmarks and Statistical Report 2013
Organizational Culture & Evaluation6
Does your library measure outcomes or impacts of teen programming?
% Yes
42.9%
All 46.7%
100,000+ 38.1%
50,000<100,000
40.0%
15,000<50,000 46.4%
5,000<15,000 75.0%
<5,000 42.9%
Less than 50% of the libraries reported measuring outcomes or impacts of teen programming, althoughseveral libraries did note that they are planning to implement outcome-based measurement in the future.
Q: Are you satisfied with the methods used by your library to evaluate teen programs?
Organizational Culture & Evaluation7
• It would be helpful to have a clearer idea of what a youth program should be i.e. what is the purpose, what is the desire outcome.
• It is not consistent• We do not evaluate
anything past attendance.• I would like to find out
what the participant got out of program
• Difficult to measure anecdotal reports
• Evaluation is based only on circulation and attendance performance indicators
Q: Are outcomes/impacts measured in other parts of your library?
Organizational Culture & Evaluation8
• Lack of staff capacity• Do not have anyone trained
in outcome/impact measurement
• Probably the best answer for why it hasn't happened yet is "too time consuming."
• Lack of time.• Not currently considered a
priority by our governing body
• Assessing outcomes falls short of our priority list.
• No formal criteria which is implemented system-wide
Q: You indicated that you do not assess outcomes/impacts of teen programs. Why not? Check all that apply.
Organizational Culture & Evaluation9
Q: Do the agencies that fund your teen programs require that you report outcome/impact measures?
Organizational Culture & Evaluation10
Organizational Culture and Impact
Organizational Culture & Evaluation11
Sub-cultures are currently the greatest inculcator of values (Schein, 1992) regarding: 1.the worth and purpose of evaluations,2.postulation of negative influence, 3.evaluation seen as “busy work”
Sub-Culture as Inhibitor
Organizational Culture & Evaluation12
Common Responses from Follow-up Interviews
Seen as “busy-work” Rely on informal feedback Some staff value the process,
some question it, seen as additional work
Against staff comfort level Comments are filtered –each
manager chooses Staff wouldn’t want it (new
system) to be more onerous on them
Staff question why, “Are you evaluating me?”
Inhibitors
Organizational Culture & Evaluation13
Lack of education, not able to produce meaningful results and change
Lack of inclusion: “Big picture” relevanceRequired training in best practicesLibrarians feel skill set is inadequate to the
task, so they are reluctant to engage – (Similar circumstances to the introduction of technology in mid to late 1990s in public libraries (Author’s observation))
Technological impasse Consider the law of unintended consequences
(how can/does it apply here?)
Sustainable change requires cultural focusing (1)
Organizational Culture & Evaluation14
"Cultural change is one of the most important factors to consider but one of the hardest to implement" (Preston, 2004).
“In general, we found a number of library staff skeptical of quantitative or qualitative data from customers, preferring instead to rely on their own assumptions and past practices to make decisions.” (Hiller et al, 2008)
“Due to the ‘everyone does the same thing’ culture and operational model it was impossible to make a change to workflow in one area without it directly impacting other areas - therefore systemic change was necessary.” (Nussbaumer & Merkley, 2012)
Sustainable change requires cultural focusing (2)
Organizational Culture & Evaluation15
Integration of Culture & Evaluation Model (1)
Organizational Culture & Evaluation16
The following model was developed to demonstrate the steps an organization goes through in regards to moving from an input based to an outcome based evaluation system.
It highlights the operational and cultural transformation process.
This model can be used to determine where an organization is situated (at what stage) in terms of its use of outcome evaluation.
The model is designed to serve as a “roadmap”, to assist organizations in their successful movement through a series of steps, by infusing a culture of outcome evaluation.
Once an organization has achieved stage 4, evaluation then serves as a tool for: realizing organizational priorities, educating stakeholders and funders of the range of quantitative and qualitative program and service impacts, and identifying operational issues and challenges.
Integration of Culture & Evaluation Model (2)
17
Creating an outcome evaluation culture
Integration of Culture & Evaluation Model (3)
Organizational Culture & Evaluation18
Kotter (2008) argues that getting buy-in is not enough because it only engages the head, not the heart.
Moving through the stages is akin to Kotter’s step 6, creating short term wins, building staff confidence, an incremental approach
Somewhere in the transition between step 1 to 2 need to create and articulate a vision/mission statement. Why do we evaluate? What is it all about?
Building cultural assessment seen as a strategic priority – otherwise over shadowed by more “important work”
“When faculty and staff perceive that the administrators are only motivated to create a culture of assessment for accreditation, cynicism and low-motivation will likely result” (Lakos and Phipps, 2004).
Next Steps
Organizational Culture & Evaluation19
Formalize the Integration of Culture & Evaluation Model
Create an organizational evaluation audit to:Assess the stage of the model an organization is at
Develop strategies to assist moving it forward
Create a sustainability approach
Design and deliver the next phase of the research
Next Steps
Q: Do the agencies that fund your teen programs require that you report outcome/impact measures?
Organizational Culture & Evaluation20
Ontario-wide survey to elected municipal officials focused on understanding what information they consider when in their deliberations on financial support to libraries’ budgets.
In part this research will provide insight for libraries in how to structure, and communication the results of, their institutional evaluations.