Collection Analysis:Collection Analysis:OverviewOverview
Sponsored by ALCTS CMDS Measures & Education Sponsored by ALCTS CMDS Measures & Education CommitteesCommittees
Peggy Johnson,Peggy Johnson,Associate University LibrarianAssociate University Librarian
University of MinnesotaUniversity of [email protected]@umn.edu
““Culture of Assessment/Evaluation”Culture of Assessment/Evaluation”
Way to demonstrate Way to demonstrate • RelevanceRelevance• ValueValue• ImpactImpact
Considered from the view ofConsidered from the view of• Users Users • Stakeholders Stakeholders
Amos Lakos & Shelley Phipps – “Culture of Assessment”Amos Lakos & Shelley Phipps – “Culture of Assessment”John Crawford – “Culture of Evaluation”John Crawford – “Culture of Evaluation”
““Those who fail to move in Those who fail to move in the direction of the direction of systematic systematic assessmentassessment will be unable will be unable to cope with the to cope with the increasingly difficult increasingly difficult questions that promise to questions that promise to confront collection officers confront collection officers in years to come.”in years to come.”
Mark Sandler, Univ. of MIMark Sandler, Univ. of MI
Why do we do it?Why do we do it?
As part of good management. As part of good management. Accountability: To demonstrate to funders Accountability: To demonstrate to funders
and clients that the service is delivering and clients that the service is delivering the benefits expected when the the benefits expected when the investment was madeinvestment was made
Decision-making: To ensure that resources Decision-making: To ensure that resources are being used efficiently and effectively are being used efficiently and effectively (an internal control mechanism)(an internal control mechanism)
Marketing: To report success and Marketing: To report success and accomplishments (public relations)accomplishments (public relations)
Collection assessment assumes Collection assessment assumes that the criteria for success are that the criteria for success are defined and understood by those defined and understood by those doing the assessment and those doing the assessment and those to whom it is being reports.to whom it is being reports.
What is it?What is it?
A mechanism to determine:A mechanism to determine: If the collection is meeting its objectivesIf the collection is meeting its objectives How well it is serving its usersHow well it is serving its users In which ways or areas it is deficient, and In which ways or areas it is deficient, and
what remains to done to develop the what remains to done to develop the collectioncollection
If selectors are performing their If selectors are performing their responsibilities effectivelyresponsibilities effectively
How to allocate collections/access fundsHow to allocate collections/access funds
How is assessment different How is assessment different from evaluation?from evaluation?
EvaluationEvaluation determines how well the determines how well the collection supports the goals, needs, collection supports the goals, needs, and curriculum of the parent and curriculum of the parent organization. organization.
Assessment Assessment examines or describes examines or describes collections either in their own terms collections either in their own terms or relative to other collections and or relative to other collections and checklists. checklists.
Who is the audience ?Who is the audience ?
Accreditation agenciesAccreditation agencies Parent organization (administration, Parent organization (administration,
board, senior management)board, senior management) Library administrationLibrary administration CDM supervisorCDM supervisor SelectorSelector User community or communities User community or communities Consortial partnersConsortial partners
How can we do it well?How can we do it well?
SimpleSimple PracticalPractical RepeatableRepeatable Clear focusClear focus Understandable resultsUnderstandable results Meaningful resultsMeaningful results Results lead to actionResults lead to action
Collection-based Measures Look at:Collection-based Measures Look at:
SizeSize
GrowthGrowth
Coverage (depth, breadth, balance)Coverage (depth, breadth, balance)
Collection-based MeasuresCollection-based Measures
Checking lists, catalogs, Checking lists, catalogs,
bibliographiesbibliographies
Evaluating the collection directlyEvaluating the collection directly
Compiling comparative statisticsCompiling comparative statistics
Application of collection standardsApplication of collection standards
Use- and User-based Measures Look at:Use- and User-based Measures Look at:
Who is using the collection?Who is using the collection?
How often?How often?
What are users’ expectations?What are users’ expectations?
What are user’s needs?What are user’s needs?
What are their perceptions?What are their perceptions?
Use- & User-based MeasuresUse- & User-based Measures
Circulation studiesCirculation studies
In-house use studiesIn-house use studies
Survey of usersSurvey of users
Shelf availability studiesShelf availability studies
Analysis of online usage of electronic resourcesAnalysis of online usage of electronic resources
Analysis of ILL statisticsAnalysis of ILL statistics
Citation studiesCitation studies
Document delivery testsDocument delivery tests
Cost-per-use Cost-per-use
Quantitative MeasuresQuantitative Measures
Count thingsCount things• UseUse• ExpendituresExpenditures• TitlesTitles• Physical itemsPhysical items
Quantitative MeasuresQuantitative Measures
TitlesTitles
Circulation transactionsCirculation transactions
ExpendituresExpenditures
E-metricsE-metrics
ILL transactionsILL transactions
Ratios (monographs/serials; volumes/students; Ratios (monographs/serials; volumes/students;
expenditures/degree programs; electronic/print)expenditures/degree programs; electronic/print)
E-MetricsE-Metrics
Online sessionsOnline sessions Documents downloadedDocuments downloaded Records downloadedRecords downloaded Virtual visitsVirtual visits Turn-awaysTurn-aways Alert usageAlert usage Personal profile usersPersonal profile users Remote versus onsite usageRemote versus onsite usage
Qualitative ResearchQualitative Research
““A process of inquiry that draws data A process of inquiry that draws data from the context in which events from the context in which events occurs . . . using induction to derive occurs . . . using induction to derive possible explanations based on possible explanations based on observed phenomena”observed phenomena”
Gorman and Clayton, Gorman and Clayton, Qualitative Research for the Qualitative Research for the Information Professional: A Practical HandbookInformation Professional: A Practical Handbook, , 22ndnd ed. (London: Facet, 2005) ed. (London: Facet, 2005)
Qualitative Measures Look at:Qualitative Measures Look at:
StrengthsStrengths
WeaknessesWeaknesses
Non-strengthsNon-strengths
Qualitative MeasuresQualitative Measures
Provide the contextProvide the context Offer a way to understand the Offer a way to understand the
attitudes that inform the statisticsattitudes that inform the statistics
Quantitative MeasuresQuantitative Measures
Focus groupsFocus groups Online or printed surveysOnline or printed surveys Interviews (structured or Interviews (structured or
unstructured) unstructured) ObservationObservation
Collection Analysis MethodsCollection Analysis MethodsUse- or User-basedUse- or User-based Collection-basedCollection-based
QuantitativeQuantitative •ILL statisticsILL statistics•Circulation statisticsCirculation statistics•In-house use statisticsIn-house use statistics•Document delivery Document delivery statisticsstatistics•Shelf availability Shelf availability statisticsstatistics•E-metricsE-metrics
•Collection size and Collection size and growthgrowth•Materials budget size and Materials budget size and growthgrowth•Collection size standards Collection size standards and formulasand formulas•Expenditures by subjectExpenditures by subject•RatiosRatios
QualitativeQualitative User opinion surveysUser opinion surveysUser observationUser observationFocus groupsFocus groupsInterviewsInterviews
List checkingList checkingVerification studiesVerification studiesCitation analysisCitation analysisDirect collection checkingDirect collection checkingCollection mapping Collection mapping (assigning conspectus (assigning conspectus levels)levels)
Where to start?Where to start?
Define the question or problemDefine the question or problem Determine metrics to useDetermine metrics to use Decide:Decide:
• Where to locate the informationWhere to locate the information• Who will collect the informationWho will collect the information• Who will analyze and report the Who will analyze and report the
informationinformation• Who will act on the informationWho will act on the information
RememberRemember
Chose measures that matterChose measures that matter Chose an approach that is simpleChose an approach that is simple Don’t aim for perfection—good ‘nuff Don’t aim for perfection—good ‘nuff
is OKis OK Don’t do it once and never againDon’t do it once and never again Know your audienceKnow your audience Present data in a context—explain Present data in a context—explain
what it meanswhat it means