preparing to work effectively with libqual+ ® survey results ala annual conference washington, dc...
TRANSCRIPT
Preparing to Work Effectively withLibQUAL+® Survey Results
ALA Annual ConferenceWashington, DC
June 26, 2007
Raynna Bowlby
Martha Kyrillidou
Association of Research Libraries
old.libqual.org
old.libqual.org
Preparing to Work Effectively with Preparing to Work Effectively with LibQUAL+® Survey Results Survey Results
Prepared:
properly expectant, organized, or equipped; ready
Ready:
completely prepared or in condition for immediate action or use or progress
Dictionary.com Unabridged (v 1.1)
old.libqual.org
Desired OutcomesDesired OutcomesAs a result of our work together today, you will be prepared to…As a result of our work together today, you will be prepared to…
• Perform some simple analyses of the LibQUAL+® survey results data
• Today we’ll be working mainly with the 22 core LibQUAL+® questions and Comments
• Organize your library colleagues and groups / teams to work with LibQUAL+® results
• Present the results to different stakeholders• Utilize the data to target areas for improvement• Begin a process of continuous assessment
• Thank you for helping us “beta test” this workshop; we welcome your feedback!
old.libqual.org
After the After the LibQUAL+LibQUAL+®® Survey -- Work PlanSurvey -- Work Plan
• Understanding the Data and Results– Interpreting representativeness, zone of tolerance, peer comparisons,
longitudinal analysis, analytics and SPSS
• Developing the Organization– Confirming survey objectives, clarifying roles & responsibilities, presenting
results, engaging others in utilizing data, discussing results & implications
• Using LibQUAL+® Effectively– Targeting incremental improvements, applying data in decision-making,
implementing a cycle of continuous assessment and improvement
old.libqual.org
After the After the LibQUAL+LibQUAL+®® Survey -- Work PlanSurvey -- Work Plan
• A list of steps describing work that can be accomplished after receiving the survey, including quantitative and qualitative data analysis and organizational development
• Think about:– Which steps might your library undertake?– Will the assignment be individual or a group / team?– What KSAs might best support the accomplishment of
this work?– Who will be assigned?
– (Who will you consult with to make these decisions)?
old.libqual.org
Understanding the Data & ResultsUnderstanding the Data & Results
old.libqual.org
Results Notebooks
• Sections for Overall, Undergraduates, Graduates, Faculty, Staff, Library Staff include:– Demographic Summary– Core Questions Summary– Dimensions Summary– Local Questions– General Satisfaction Questions– Information Literacy Outcomes Questions– Library Use Summary
• Appendix describing changes in the dimensions and the questions included in each dimension.
old.libqual.org
ResultsResults
• Results Notebooks (PDF)http://old.libqual.org/Manage/Results/index.cfm
– Group Notebook (PDF)
– Unified Notebook (PDF)
– User Comments
– Excel Data File
SPSS data files (if requested at registration) will be e-mailed to the primary contact two to three months after the session has closed.
old.libqual.org
DO YOU UNDERSTAND DO YOU UNDERSTAND YOUR RESULTS?YOUR RESULTS?
old.libqual.org
Representativeness - Standard Disciplines: Representativeness - Standard Disciplines: An Academic LibraryAn Academic Library
Respondents
Population
old.libqual.org
Representativeness - Customized Disciplines: Representativeness - Customized Disciplines: An Academic LibraryAn Academic Library
Respondents
Population
old.libqual.org
Radar Chart BasicsRadar Chart Basics
old.libqual.org
Key to Radar ChartsKey to Radar Charts
old.libqual.org
Understanding a Radar ChartUnderstanding a Radar Chart
old.libqual.org
What Do the Colors Mean?What Do the Colors Mean?
GreenAnd
Blue
old.libqual.org
What Do the Colors Mean?What Do the Colors Mean?
Little Yellow
old.libqual.org
What Do the Colors Mean?What Do the Colors Mean?
A lot of Yellow
old.libqual.org
What Do the Colors Mean?What Do the Colors Mean?
Red
old.libqual.org
Dimension SummaryDimension Summary
old.libqual.org
Understanding Your Individual Results: Understanding Your Individual Results: Thermometer ChartsThermometer Charts
Key Term: Zone of Tolerance
old.libqual.org
Dimension SummaryDimension Summary
Key Term: Zone of Tolerance
old.libqual.org
Peer ComparisonsPeer ComparisonsLibQUAL+ 2006
Faculty Ratings of Journal CollectionsARL Libraries
5.00
6.00
7.00
8.00
9.00
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37
Red Square = Perceived Service Performance
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
UVA
Source: Jim Self, University of Virginia, Presented at Performance Measurement in Academic Libraries Workshop, EBLIP4, Durham, North Carolina, May 11, 2007
old.libqual.org
Peer ComparisonsPeer ComparisonsLibQUAL+ 2006
Overall Quality of the Service Provided by the Library38 ARL Libraries
7.87
7.637.63
6.51
6.61
5.87
7.487.52
7.87
5.5
Undergraduates
Graduates
Faculty
UVA UVA
UVA
Source: Jim Self, University of Virginia, Presented at Performance Measurement in Academic Libraries Workshop, EBLIP4, Durham, North Carolina, May 11, 2007
old.libqual.org
Peer ComparisonsPeer Comparisons
Adequacy GapThe difference between the minimum and perceived score.
-0.40
-0.20
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
UT Austin
ARL
Peers
Source: Fred Heath, LibQUAL+™ Results Meeting, ALA Annual Conference, Seattle, WA, January 22, 2007
old.libqual.org
Longitudinal AnalysisLongitudinal Analysis
Source: Fred Heath, LibQUAL+™ Results Meeting, ALA Annual Conference, Seattle, WA, January 22, 2007
old.libqual.org
NORMS
• Identify your score
• Compare it to a relevant norms table– Year– Subgroup– Dimension
• Norms are stable and are not calculated on an annual basis anymore (see article)
old.libqual.org
ANALYTICS
• Institution Explorer– User group and Discipline analysis
• Representativeness graphs• Radar chart• Library Use chart• Thermometer chart• Cumulative percentile distribution
• Longitudinal
old.libqual.org
SPSS
• What is SPSS?– Statistical Package for the Social Sciences
• How do I know that my data are trustworthy?– Reliability and Validity Analysis
• Are there differences between different groups?– Compare mean scores for subgroups
old.libqual.org
Quantitative Analysis: Excel DataQuantitative Analysis: Excel Data
• Excel data files available on the LibQUAL+® Web sitehttp://old.libqual.org/Manage/Results/index.cfm
• Use customized radar chart template to create custom analyseshttp://old.libqual.org/Manage/Resources/
SampleRadarChart/index.cfm
old.libqual.org
Qualitative Analysis: User CommentsQualitative Analysis: User Comments
• About one-half of users include comments on their surveys
• User Comments available on the LibQUAL+® Web site– Download comments in Excel or text file
• Skim the comments
• Conduct ATLAS.ti analysis
old.libqual.org
Identifying What is ActionableIdentifying What is Actionable
• Conclusions and recommendations are grounded in survey data
Identifying What is Actionable Worksheets:
Priorities, Weaknesses, Strengths
old.libqual.org
Developing the OrganizationDeveloping the Organization
old.libqual.org
Confirming Survey Confirming Survey Objectives/Desired OutcomesObjectives/Desired Outcomes
• Conducting LibQUAL+® is not an end in itself• Articulate WHY the library wants feedback from
users; be specific• Indicate potential targets of interest, for example:
– Customer input for library Strategic Planning– Assessment for institutional accreditation– Intention to become more user-centered– Focus on specific institutional or library strategic
initiatives such as space/facilities issues, collection(s) changes, web re/design, customer service, etc.
old.libqual.org
Identifying & Clarifying Identifying & Clarifying Roles & ResponsibilitiesRoles & Responsibilities
• Develop an understanding of the work that your library may want to undertake after the LibQUAL+® survey (see the Work Plan)
• Decide what work your library will/will not do• Determine if your library will assign the
responsibility to a single individual or will assemble a group / team
• Identify the knowledge, skills, abilities (KSAs) needed to successfully perform the desired work
• Assign the work to match the KSAs and/or acquire the needed KSAs
old.libqual.org
Knowledge, Skills, & Abilities (KSAs)Knowledge, Skills, & Abilities (KSAs)
• Commitment to focus on user-centered outcomes & to apply user-centered thinking
• Dedication to achieving continuous process improvement
• Ability to think & act system-wide, to see the “big picture”
• Effective communication skills• Comfortable w/statistics & data• Critical thinking / analytical skills
old.libqual.org
Knowledge, Skills, & Abilities (KSAs)Knowledge, Skills, & Abilities (KSAs)Useful to have or to develop (individually or collectively)Useful to have or to develop (individually or collectively)
• Statistical methods & quantitative data analysis (e.g., use of SPSS)
• Qualitative data analysis (e.g., Atlas-TI)• Automate data collection tasks (e.g., write scripts)• Design charts & graphs (e.g., Excel)• Writing (e.g., Word)• Presentation (e.g., Powerpoint)
old.libqual.org
Implementing a Implementing a LibQUAL+LibQUAL+®® Group or Team Group or Team
Consider the purpose / scope of the group’s work:– Broad, all assessment vs. specific,
LibQUAL+® only– On-going vs. situational?– Task-oriented, e.g. administer, analyze?– Facilitative, e.g. communicate, present, assist
others in understanding/utilizing data/results?– Leadership-focused, e.g. synthesize,
recommend, direct others to implement changes?
old.libqual.org
Components of the ChargeComponents of the Chargefor a Group/Teamfor a Group/Team
• Purpose and/or Scope• Products / Deliverables• Assumptions and Parameters
– the extent of the group’s autonomy / authority– approval process(es)– w/whom and how often the group will communicate
w/colleagues– suggested approaches for methods of operation and
how the group will function– available resources (budget, people, etc.)
• Timeline• Member KSAs
old.libqual.org
Assessment Group ResourcesAssessment Group Resources
• Coming soon: ARL SPEC Kit on Assessment– Stephanie Wright (U. Washington) and Lynda White (U. Virginia)
• Boston College: http://www.bc.edu/libraries/about/assessment/• Ohio State University: http://library.osu.edu/sites/staff/assess/• U. Colorado (Boulder):
http://ucblibraries.colorado.edu/internal/assessment/index.htm• U. Connecticut:
http://www.lib.uconn.edu/about/administration/surveys/• U. Georgia: http://dataserv.libs.uga.edu/assessment/index.html• U. Kansas:
http://www.informationservices.ku.edu/assessment/charge.shtml• U. Maryland: http://www.lib.umd.edu/PASD/MIS/larc/• U. Mass: http://www.library.umass.edu/assessment/index.html• U. Washington: http://www.lib.washington.edu/assessment/• Washington State U.:
http://www.wsulibs.wsu.edu/general/WG/AWG.html
old.libqual.org
Presenting Results to StakeholdersPresenting Results to Stakeholders
• Identify all of the stakeholders or constituents who want and need to know about the survey results
• Consider the “stake” of each of the above; what specific aspect of LibQUAL+® will be of most interest / concern
• Determine how to communicate with each identified stakeholder
old.libqual.org
Brown University LibraryBrown University LibraryStakeholder Accountability MatrixStakeholder Accountability Matrix
http://www.arl.org/arldocs/stats/statsevents/laconf/2006/BowbyOMahony.ppthttp://www.arl.org/arldocs/stats/statsevents/laconf/2006/BowbyOMahony.ppt
Stakeholder "Stake"Relationship &
Communication with the Library
Indicators
ProvostEnsure PAE progress & success;
ensure effective Library management & utilization of
resources, and alignment with PAE
Inform indirectly through OIR; followed up by direct dialogue with
University LibrarianAll key indicators at broad level
University Resources Committee
Determine allocation/reallocation of financial resources
Inform directly via annual written budget request and oral presentation
Interested in indicators at more detailed level, particularly those
related to financial aspects of the Library and measures related to
specific PAE goals such as collections and space/facilities
Office of Institutional
Research
Ensure that indicators measure PAE progress
Inform directly; work collaboratively to identify, develop, and refine
appropriate measures; OIR assists in analyses & presentation of PAE
indicators
All key indicators at broad level
Corporation Committee on
Academic Affairs
Determine academic priorities & programs, with recommendations to
the full Corporation; direct the President, Provost, and others to
enact
Inform indirectly via Provost; CCAA receives regular reports from the
University administration containing assessment information on agreed-upon measures of academic quality
All key indicators at broad level
Library Advisory Council
Advise the President, annual report; chair’s influence on CCAA
Inform directly, 2-3 times per year; opportunity to build understanding
ARL size data, especially expenditures; LibQUAL+
satisfaction, especially students; space/facilities
old.libqual.org
Assumption College Assumption College LibQUAL+LibQUAL+®® Marketing Plan: Marketing Plan:old.libqual.org/documents/admin/ThistleLibQUAL+IntroSeattle.pptold.libqual.org/documents/admin/ThistleLibQUAL+IntroSeattle.ppt
LibQUAL Marketing Theme: Can We Hear You Now?
Dates [2/6-2/26 (3/3)] Notes Owner Done
Library Web site 1/13/06 informational page on the survey linked to from library front page Carole X
January library newsletter 1/23/06 article on the survey - who, what, where, and WHY - WIIFM Dawn X
Public relations office 1/23/06 Heidi - send her survey link for her newsletter Carole X
Academic Council 1/24/06 make the announcement at meeting Carole X
Administrator's meeting 1/24/06 make the announcement at meeting Laurie X
SGA 1/29/06 make the announcement at meeting Dawn X
Bulletin board 1/30/06 Dawn X
Communication with CE & Grad 1/30/06 Dawn X
Faculty Senate 1/30/06 make the announcement at meeting Janice X
Letter 1/30/06 faculty, admin, staff - see LibQUAL mktg. booklet (students receive email) Dawn X
Poster display 1/30/06 large and small posters - pic of Larry holding the iPOD (?) Carole X
Purchase Prizes 1/30/06 iPod, gift certificates (Nina-to get Amazon one)Dawn & Laurie x
Mobiles 2/1/06 mobiles with Larry Dawn X
Tent signs 2/1/06 library Liz X
Connection 2/3/06 survey announcement John X
Public access computers 2/3/06 wallpaper screen - posters
Laurie/ Carole/ Dawn X
Email 2/6/06 take the survey (ask Dawn to send to CE) Carole
Postcards 2/6/06 Can we Hear You Now? - pic of iPod w/ info on back Carole X
Student postcards 2/6/06 iPod pic and message on back Carole X
Announce at BI sessions 2/6/06, ongoing librarian responsible for making announcements All x
PowerPoint at BI sessions 2/6/06, ongoing Carole will create and presentation would run in a loop Carole x
Provoc article 2/8/06 article written by staff writier Dawn X
old.libqual.org
Communicate with your CustomersCommunicate with your Customers(faculty, students, other)(faculty, students, other)
• Particularly those whom you asked to participate in the survey
• Announce incentive award winners• Inform users of highlights of survey results• More importantly, what the library intends to do
“You asked for it…”
old.libqual.org
Engaging Others in Engaging Others in Understanding & Utilizing Data & ResultsUnderstanding & Utilizing Data & Results
• Spread knowledge about LibQUAL+®
• Develop a culture of accountability for listening to customers and acting on user feedback
old.libqual.org
Developing a Culture of AssessmentDeveloping a Culture of AssessmentToolkit: http://personal.anderson.ucla.edu/amos.lakos/CUtoolkit.htmlToolkit: http://personal.anderson.ucla.edu/amos.lakos/CUtoolkit.html
old.libqual.org
Spreading Knowledge about Spreading Knowledge about LibQUAL+LibQUAL+®® throughout the Library Staffthroughout the Library Staff
• Post Report on library web• Make staff aware of LQ tutorial:
– http://old.libqual.org/Information/Tools/index.cfm
• All-staff presentation• Disseminate Comments to depts, units, groups,
branches, librarians, etc.• Work w/department leaders, managers
– Identifying Library Departments with Contribution or Impact on LibQUAL+® Questions
• Other?
old.libqual.org
Corroborating Corroborating LibQUAL+LibQUAL+®®
Results w/Other DataResults w/Other Data• Consider if your LibQUAL+® data confirms your
prior expectations and results from other assessment and measurement, and, vice versa
• For example:– Undergrads have increasing needs & desires re:
Library as Place and show increases in library gate counts/attendance
– Faculty & grad students have access from home or office as highest priority and library web logs demonstrate increasing use from campus depts.
old.libqual.org
Discussing Results & ImplicationsDiscussing Results & Implications
• Go beyond informing…
• Lead efforts to analyze, synthesize, and interpret findings
old.libqual.org
Brown University Library:Brown University Library:Using the “Mini-Survey” to Explain Using the “Mini-Survey” to Explain LQ+LQ+®® to Stakeholders to Stakeholders
old.libqual.org
The Library Summit:The Library Summit:Clemson University & U. of Texas at AustinClemson University & U. of Texas at Austin
http://old.libqual.org/documents/admin/LibQUAL_Summit.pdfhttp://old.libqual.org/documents/admin/LibQUAL_Summit.pdf
old.libqual.org
Why Hold a Library Summit?Why Hold a Library Summit?http://www.lib.utexas.edu/vprovost/assessment/libsummit/pdf/info_pkt.pdfhttp://www.lib.utexas.edu/vprovost/assessment/libsummit/pdf/info_pkt.pdf
old.libqual.org
Identify 2 or 3 LibQUAL+® questions for
which you/your department has primary
responsibility
Do these “cluster” in a specific dimension?
Being User-Centered:Being User-Centered:Guiding Library Staff in Understanding & Using Guiding Library Staff in Understanding & Using LibQUAL+LibQUAL+®® Results Results
old.libqual.org
“We know what’s best”
“They are wrong”
“We don’t have the resources to…”
““...only customers judge ...only customers judge quality; all other quality; all other judgments are judgments are essentially irrelevant.”essentially irrelevant.”
Zeithaml, Parasuraman, Berry.(1999). Delivering Quality Service. NY: The Free Press
Overcoming Resistance & Barriers:Overcoming Resistance & Barriers:Changing Attitudes & BehaviorsChanging Attitudes & Behaviors
old.libqual.org
Individual Exercise:Individual Exercise:After the After the LibQUAL+LibQUAL+®® Survey -- Work PlanSurvey -- Work Plan
• Review the list of steps describing work that can be accomplished after receiving the survey, including quantitative and qualitative data analysis and organizational development
– Which steps might your library undertake?– Will the assignment be individual or a group / team?– What KSAs might best support the accomplishment of
this work?– Who will be assigned?
– (Who will you consult with to make these decisions)?
old.libqual.org
Using Using LibQUAL+LibQUAL+®® Effectively Effectively
old.libqual.org
Targeting Incremental ImprovementsTargeting Incremental Improvements
• From all of the data, determine what can and should be addressed
• Prioritize some action items– Align with mission, vision and goals of parent
organization– Address users’ top priorities, by user group– Improve areas of strong user dissatisfaction– Build on strengths, if they are truly user needs
and priorities– Identify work that can be de-emphasized and
resources that can be reallocated
old.libqual.org
S.M.A.R.T. GoalsS.M.A.R.T. Goals
• S Specific– the desired outcome or result is clearly defined
• M Measurable– accomplishment can be charted and/or observed
• A Attainable– achievable, goal is challenging but realistic
• R Relevant– results-oriented, in line with institutional goals and library vision
• T Timely– deadlines are set for accomplishment
old.libqual.org
SMART Goals -- ExamplesSMART Goals -- Examples
• 75% of materials acquired from other libraries are received by users within 10 days of request by 12/07
• 50% of new books are on shelf within 5 days after library receipt by 10/07
• 60% courses utilizing WebCT include links to library research materials by 3/08
• Lower the unit cost of each service desk transaction by 10% from FY’08 to FY’08
Versus general goals:Improve ILL turn-around timeMake new materials accessible to users more quicklyDecrease staffing budget
old.libqual.org
old.libqual.org
Applying Data in Decision-MakingApplying Data in Decision-Making
• Assessment and LibQUAL+® data and results are easily accessible and present when, where, and with whom decisions are made in your library
• Goals, priorities, and resource allocation are influenced by customer needs and wants
old.libqual.org
Archiving Data for AccessArchiving Data for Access
old.libqual.org
Archiving Data for AccessArchiving Data for Access
old.libqual.org
Cycle of Planning and AssessmentCycle of Planning and Assessment
www.mindtools.com
old.libqual.org
Identifying & Using Best PracticesIdentifying & Using Best Practices
• Seeking out and learning from colleague institutions– Participate in LibQUAL+® as part of a
consortial, regional, or interest-group implementation
– Identify peer institutions who may agree to do LibQUAL+® during the same cycle
• Implementing improvements based upon best practices
old.libqual.org
Integrating LibQUAL+® w/ Other Assessments
• Confirming your conclusions• Obtaining user specifications for initiatives• Monitoring progress• Some options:
– Suggestions boxes– Focus groups– Exit surveys
old.libqual.org
Planning Next Steps:Planning Next Steps:Continuous Assessment & ImprovementContinuous Assessment & Improvement
old.libqual.org
In development ….
Customized Consulting Services:– Longitudinal data analysis, across multi years– Tailored peer comparisons, libraries you choose– Assistance w/ content analysis for Comments– Facilitation of LibQUAL+® staff retreat or summit– Guidance in preparing communication plan– Customized training in assessment skills– Launch an assessment group or team– Assistance in identifying key customer issues for future
action
old.libqual.org
LibQUAL+LibQUAL+®® Resources Resources
• LibQUAL+® Web site:http://old.libqual.org
• Publications:http://old.libqual.org/publications
• Events and Training:http://old.libqual.org/events
• Gap Theory/Radar Graph Introduction:http://old.libqual.org/Information/Tools/libqualpresentation.cfm
• LibQUAL+® Procedures Manual:http://old.libqual.org/Publications/index.cfm
old.libqual.org
LibQUAL+LibQUAL+®® Team Team
• MaShana DavisTechnical Communications [email protected]
• Yolanda GlassAdministrative [email protected]
• Kristina JusthCustomer Relations [email protected]
• Martha KyrillidouDirector, Statistics and Service Quality [email protected]
• Khyati NayakSenior Applications [email protected]
• Gary RoebuckTechnical Operations [email protected]