analyzing data, getting results: making it all make sense
Post on 13-Dec-2014
92 Views
Preview:
DESCRIPTION
TRANSCRIPT
Analyzing Data, Getting ResultsMaking it All Make SenseJenn RileyUniversity of North Carolina at Chapel Hill
SCEL
C Re
sear
ch D
ay3/
5/13
2
Evidence-driven decisions are a
powerful guide for library operations.
SCEL
C Re
sear
ch D
ay“The plural of anecdote is not data.”
After a quote with the opposite meaning, by Raymond Wolfiger.
3/5/
13
3Sometimes attributed to Frank Kotsonis.
SCEL
C Re
sear
ch D
ay
“There are three kinds of lies – lies, damned lies, and statistics.”
Mark Twain, perhaps after Benjamin Disraeli.
3/5/
13
4
SCEL
C Re
sear
ch D
ay
Using data for planning library operations
3/5/
13
5
Existence/hours of service points
Materials to buy/license/accept/digitize/keep/preser
ve
Designing web sites and other online
resources
Effectiveness of/satisfaction with procedures/services
Evaluating a pilot service or project
Projecting future expenditures
SCEL
C Re
sear
ch D
ay
Both cost and value are keyALCTS Heads of Technical Services in Large Research Libraries Interest Group, Task Force on Cost/Value Assessment of Bibliographic Control (2010)
Proposes definitions of value for cataloging:
3/5/
13
6
Discovery successUseDisplay understandingData interoperability
Support for FRBR user tasksThroughput/timelinessSupport administrative goals
SCEL
C Re
sear
ch D
ay
Example studies
•By Joyce Chapman, then at North Carolina State University• Benefits of manually enhanced metadata
for images• Comparing effort to utility for specific
EAD elements
3/5/
13
7
See Chapman, Joyce. “Metrics & Management: Cost & value of metadata workflows.” SAA 2011. http://www.academia.edu/1708422/Return_on_Investment_Metadata_metrics_and_management
SCEL
C Re
sear
ch D
ay
Some common analyses
3/5/
13
8
Cost per unit produced
Change over time
Error/problem rate
Predicting impact of a change
Identifying unmet needs
SCEL
C Re
sear
ch D
ay
Back to library scenarios
3/5/
13
9
SCEL
C Re
sear
ch D
ay
Existence/hours of service points• Who is using what and when?• How can we most effectively staff them?• Costs• Staff time• Facilities management costs
• Benefits• Number and type of visitors, and how they use it• Service transactions completed• Specific services used at the location
• Other data to collect• Usage by time of day
• Calculate cost per transaction
3/5/
13
10
SCEL
C Re
sear
ch D
ay
Materials to buy/license/accept/digitize/keep/preserve• Should we acquire, make more accessible, or keep this?• Costs• Initial purchase/license• Ongoing license/maintenance• Staff for
cataloging/processing/digitizing/ingesting/preserving• Software• Hardware/storage
• Benefits• Current and predicted future use• Opportunity for transformative use
3/5/
13
11
SCEL
C Re
sear
ch D
ay
Evaluating a pilot service or project• Is the cost/benefit ratio appropriate?• What is the raw cost?• But it’s not all about cost/benefit:• Is the pilot achieving its aims?• Does this [whatever] do what we thought it
would?• What collateral effects will it have?• Were the assumptions we made correct?
• Data collection will be varied for this task
3/5/
13
12
SCEL
C Re
sear
ch D
ay
Designing web sites and other online resources• A/B testing• User-centered design• Satisfaction surveys with previous
iterations, similar sites, or prototypes•Web stats for previous iterations or similar
sites• Task-based usability testing• Don’t forget the cost of sustaining it once
you have it up!
3/5/
13
13
SCEL
C Re
sear
ch D
ay
Effectiveness of/satisfaction with procedures/services• What parts of our current service are users most and
least happy about?• What are the ineffieciences in our procedure for
[whatever]?• Some data collection ideas• User surveys• Ratio of potential to actual users• Ratio of returning to non-returning users• Error/failure rates• Time from request to delivery• Time tracking during staff activity
3/5/
13
14
SCEL
C Re
sear
ch D
ay
Projecting future expenditures• Equipment• Define its lifecycle• Amortize purchase cost• Add in maintenance costs• Compare to use as context
• Staff• Educated guess at raises, turnover, benefit costs changes
• Consider:• Inflation• Past trends• Upcoming sea changes
3/5/
13
15
SCEL
C Re
sear
ch D
ay
Strategies for getting data that can be
analyzed
3/5/
13
16
SCEL
C Re
sear
ch D
ay
Tracking use
• Circulation• COUNTER/SUSHI• Physical visitors•Web hits• Social media engagement• Attendance at events/sessions
3/5/
13
17
SCEL
C Re
sear
ch D
ay
Tracking time
• Can be effective when collected as a representative snapshot• Options for data collection• Clipboard next to a clock• Spreadsheet• Free time tracking apps•Make it as simple as possible
3/5/
13
18
SCEL
C Re
sear
ch D
ay
Calculating costs
• Staff time• 2080 hours per year is full time• Standard benefit percentages
•Materials (including software)• Initial purchase• Maintenance contracts for big-ticket items• Amortize big costs over time in service• Overhead• Universities typically have standard rates
3/5/
13
19
SCEL
C Re
sear
ch D
ay
Calculating error rates
•Both objective and subjective criteria• Typically best when done as a sample•Consider both automated and manual
means to locate errors for study
3/5/
13
20
SCEL
C Re
sear
ch D
ay
Categorization
• Putting things into like groups• Compare size of groups to one another• Compare effort spent on one group to another• Compare priority/value of one group to
another• Can be done at time of data collection, or
afterwards• Good idea to have some sense of
categories at the beginning of the study
3/5/
13
21
SCEL
C Re
sear
ch D
ay
Calculating benefit
• Change in knowledge or status• Over time• After an interaction• Survey – ask about knowledge level before
and after• Pre- and post-tests• Indirect measures• Number of people reached• Use
3/5/
13
22
SCEL
C Re
sear
ch D
ay
Additional data analysis strategies
3/5/
13
23
SCEL
C Re
sear
ch D
ay
Mechanics
• Code qualitative data to make it processable•Make sure you pick a representative and
consistent sample• Extrapolate based on known data when
you need to• ALWAYS do a sanity check• Spreadsheets are your friend
3/5/
13
24
SCEL
C Re
sear
ch D
ay
More advice
• Context is key• Don’t be paralyzed by a perceived need for
perfection• Know your basic analysis plans before you
collect/identify data• Utilize pilot projects to generate data where
there is none• Use the right tool for the job• Document your assumptions• It’s OK to use “napkin math”
3/5/
13
25
SCEL
C Re
sear
ch D
ay
Get in the habit of collecting data.
It will make your next decision easier.
3/5/
13
26
SCEL
C Re
sear
ch D
ay
Thank you!
Questions and discussion
jennriley@unc.edu
3/5/
13
27
top related