Download - Milestones in Statistical Thinking
Understanding the Methodological Understanding the Methodological Foundations of Public Library Foundations of Public Library
National Rating SystemsNational Rating Systems
Ray LyonsRay Lyons
Library Statistics for the 21st Library Statistics for the 21st Century WorldCentury World
IFLA Satellite Meeting - Montréal, QuébecIFLA Satellite Meeting - Montréal, Québec
August 18 - 19, 2008 August 18 - 19, 2008
Milestones in Statistical Thinking Milestones in Statistical Thinking
• Historical account by Alain DesrosiHistorical account by Alain Desrosièères: res:
The Politics of Large Numbers: A History of The Politics of Large Numbers: A History of Statistical Reasoning Statistical Reasoning (Harvard University Press, 1998) (Harvard University Press, 1998)
• Traces statistical ideas from 15Traces statistical ideas from 15thth century to century to present daypresent day
• Evolutionary ideas from German Evolutionary ideas from German descriptive statistics descriptive statistics and English and English political political arithmetic arithmetic pertinent to library statistical pertinent to library statistical collectioncollection
DesrosiDesrosièères describes two key statistical res describes two key statistical practices:practices:
• Creation of equivalencesCreation of equivalences−Establishing standard classifications to Establishing standard classifications to
describe phenomena by focusing on describe phenomena by focusing on similarities and ignoring differencessimilarities and ignoring differences
• Encoding Encoding −Specification and use of definitions to Specification and use of definitions to
assign individual cases to classificationsassign individual cases to classifications
Milestones in Statistical Thinking Milestones in Statistical Thinking
Purpose of this PresentationPurpose of this Presentation
To suggest that creating equivalent To suggest that creating equivalent classes and other aspects of statistical classes and other aspects of statistical data collection limit the accuracy and data collection limit the accuracy and validity of national and local public library validity of national and local public library comparisons.comparisons.
National Collection of Public Library National Collection of Public Library Statistics in the USAStatistics in the USA
• Federal-State Cooperative System (FSCS)Federal-State Cooperative System (FSCS)
• Collaborative initiated in 1980’sCollaborative initiated in 1980’s
• 1991 - First FSCS data published by U.S. 1991 - First FSCS data published by U.S. Department of Education’s National Center Department of Education’s National Center for Educational Statisticsfor Educational Statistics
National Collection of Public Library National Collection of Public Library Statistics in the USAStatistics in the USA
• Collection system renamed Public Library Collection system renamed Public Library Statistics Cooperative Statistics Cooperative (PLSC) (PLSC) in 2007in 2007
• PLSC recently transferred to Institute of PLSC recently transferred to Institute of Museum and Library Services, an agency Museum and Library Services, an agency of the federal government (www.imls.gov)of the federal government (www.imls.gov)
National Public Library Ratings National Public Library Ratings Introduced in USAIntroduced in USA
• Hennen’s American Public Library RatingsHennen’s American Public Library Ratings
(HAPLR)(HAPLR)
• Used FSCS input and output statistics for Used FSCS input and output statistics for 9000+ public libraries9000+ public libraries
• Created by library consultant Thomas Hennen Created by library consultant Thomas Hennen (library director in Wisconsin)(library director in Wisconsin)
• Issued annually since 1999Issued annually since 1999 (except for 2000 and except for 2000 and 2007)2007)
• Published (and endorsed) by American Library Association (ALA) Published (and endorsed) by American Library Association (ALA)
• Calculation methods were controversial among professionCalculation methods were controversial among profession
• Highly-rated public libraries were delighted with calculation methodsHighly-rated public libraries were delighted with calculation methods
• ““Methodologically indefensible and politically priceless” Methodologically indefensible and politically priceless” - US Library Director- US Library Director
Hennen’s American Public LibraryHennen’s American Public LibraryRatings Ratings (HAPLR)(HAPLR)
• Utilizes 5 enabling (input) indicators and Utilizes 5 enabling (input) indicators and
3 use (output) indicators3 use (output) indicators
Enabling (Input)
Total staffMaterials expendituresTotal operating expendituresNumber of printed volumesSerial subscriptions
Use (Output)
VisitsCirculation (loans)Reference transactions
Hennen’s American Public LibraryHennen’s American Public LibraryRatings Ratings (HAPLR)(HAPLR)
Hennen’s American Public LibraryHennen’s American Public LibraryRatings Ratings (HAPLR)(HAPLR)
• Recombines 8 indicators into 15 rates Recombines 8 indicators into 15 rates (ratios), similar in style to BIX(ratios), similar in style to BIX
(1)(1) Total expenditures per capitaTotal expenditures per capita
(2)(2) Materials expenditures per total expenditures Materials expenditures per total expenditures
(3)(3) Materials expenditures per capita Materials expenditures per capita
(4)(4) Staff per 1000 population Staff per 1000 population
(5)(5) Periodical subscriptions per 1000 population Periodical subscriptions per 1000 population
. . . and 10 others. . . and 10 others
• See www.haplr-index.comSee www.haplr-index.com
Study Conducted at NCLIS Study Conducted at NCLIS
• 2006 research on HAPLR ratings 2006 research on HAPLR ratings methodology overseen by U.S. National methodology overseen by U.S. National Commission for Library and Information Commission for Library and Information Science (NCLIS)Science (NCLIS)
“ “Unsettling Scores: An Evaluation of the Hennen Unsettling Scores: An Evaluation of the Hennen American Public Library Ratings, American Public Library Ratings, Public Library Public Library QuarterlyQuarterly, Volume 26, Numbers 3/4, 2007., Volume 26, Numbers 3/4, 2007.
(ISSN 016-6846)(ISSN 016-6846)
Library Journal’s Library Journal’s Public Library Public Library National Ratings AnnouncedNational Ratings Announced
• LJ LJ IndexIndex introduced June 2008 introduced June 2008**
• Rates 9200 public libraries using PLSC dataRates 9200 public libraries using PLSC data
• Co-designed with Keith Lance (Co-designed with Keith Lance (Library Research Library Research
Service,Service, State Library of Colorado)State Library of Colorado)
• Emphasizes disclosing limitations of rating Emphasizes disclosing limitations of rating methodologymethodology
Keith Curry Lance and Ray Lyons, “The New LJ Index,” Keith Curry Lance and Ray Lyons, “The New LJ Index,”
Library Journal, Library Journal, June 15, 2008, p. 38-41.June 15, 2008, p. 38-41.
Library Journal’s Library Journal’s Public Library Public Library National Ratings AnnouncedNational Ratings Announced
• Encourages responsible interpretation of Encourages responsible interpretation of rating resultsrating results
• LJ IndexLJ Index based upon 4 based upon 4 useuse (output) (output) indicators:indicators:
– VisitsVisits
– Loans (circulation)Loans (circulation)
– Internet terminal useInternet terminal use
– Program attendanceProgram attendance
Public Library Ratings UsePublic Library Ratings UseComposite ScoringComposite Scoring
• HAPLR, BIX, and HAPLR, BIX, and LJ Index LJ Index each calculate a each calculate a single composite score summarizing each single composite score summarizing each library’s performancelibrary’s performance
Circulation per capita + FTE staffing +
Subscriptions + Internet computers +
. . . + Total program attendance
Calculation Algorithm
635 (Composite Score)
Context of Comparative Library Context of Comparative Library StatisticsStatistics
• Comparative measures are:Comparative measures are:
− Measures used as part of a more general Measures used as part of a more general process to assess library value and process to assess library value and effectivenesseffectiveness
− Measures intended for use in an ongoing Measures intended for use in an ongoing process of process of performance measurementperformance measurement
Performance Measurement ModelPerformance Measurement Model
ResourcesServices Utilized
Intermediate Outcomes
End Outcomes
Outcome Measures
Outcome Measures
RESULTSEFFORTS
• Planning-for-Results Planning-for-Results approach to approach to management management
• Abandonment of established Abandonment of established operational and performance standardsoperational and performance standards
• ALA / PLA 1987 publication, ALA / PLA 1987 publication, Output Output Measures for Public Libraries: A Manual of Measures for Public Libraries: A Manual of Standardized ProceduresStandardized Procedures, , defines standard defines standard statistics and collection proceduresstatistics and collection procedures
Performance Management as Performance Management as Envisioned by the Envisioned by the
Public Library Association (PLA)Public Library Association (PLA)
Comparative Performance Comparative Performance MeasurementMeasurement
• Public sector management practice Public sector management practice • Used by state and local governments for:Used by state and local governments for:
− AccountabilityAccountability− Planning/budgetingPlanning/budgeting− Program monitoringProgram monitoring− Operational improvementOperational improvement
• Uses established standards and Uses established standards and benchmarking benchmarking (Ammons, 2001)(Ammons, 2001)
• Lack of reliable methods for identifying Lack of reliable methods for identifying peer librariespeer libraries
− Lack of instruments for measuring Lack of instruments for measuring comparabilitycomparability
• Comparisons are either approximate or Comparisons are either approximate or inaccurateinaccurate
• Can result in incorrect or misleading Can result in incorrect or misleading conclusionsconclusions
Key Problems with Library StatisticsKey Problems with Library Statistics
• National rating systemsNational rating systems use simplistic or use simplistic or imprecise criteria for identifying peers imprecise criteria for identifying peers (library type, community size, etc.)(library type, community size, etc.)
• Ignore library mission, unique community Ignore library mission, unique community needs, institutional context, etc.needs, institutional context, etc.
• Accuracy and validity of comparisons are Accuracy and validity of comparisons are compromisedcompromised
Key Problems with Library StatisticsKey Problems with Library Statistics
• Lack of criteria for evaluating measuresLack of criteria for evaluating measures
There are no ‘right’ or ‘wrong’ scores on an output measure; ‘high’ and ‘low’ values are relative. The scores must be interpreted in terms of library goals, scores on other measures, and a broad range of other factors.
- Van House, Weill, and McClure (1990)
Key Problems with Library StatisticsKey Problems with Library Statistics
FIG NEWTON COOKIES
Versus
GRANOLA BAR SNACK
100% more fruit!!!
A Diversionary ‘Visual Aid’A Diversionary ‘Visual Aid’
• National rating systemsNational rating systems apply the apply the “ “More-is-Better Rule” More-is-Better Rule”
− Views higher numbers as favorable Views higher numbers as favorable performance, lower as unfavorableperformance, lower as unfavorable
• ““More activity does not necessarily More activity does not necessarily mean better activity”mean better activity”
- Van House, Weill, and McClure (1990)- Van House, Weill, and McClure (1990)
• Striving to earn higher numbers mayStriving to earn higher numbers may compromise service qualitycompromise service quality
Key Problems with Library StatisticsKey Problems with Library Statistics
• Collection of standard statistics assumes Collection of standard statistics assumes all library resources/activities counted to all library resources/activities counted to be be equivalentequivalent
• Standardization ignores differences in:Standardization ignores differences in:
Key Problems with Library StatisticsKey Problems with Library Statistics
- Complexity
- Sophistication
- Relevance
- Quality (Merit)
- Value (Worth)
- Effectiveness
- Efficiency
- Significance
• National ratings systems add, subtract, National ratings systems add, subtract, and multiply these non-equivalent ‘units’ and multiply these non-equivalent ‘units’ of library resources, services, and of library resources, services, and products if they were equivalentproducts if they were equivalent
• Final scores imply appear arithmetically Final scores imply appear arithmetically consistent and correct even though they consistent and correct even though they count unequal unitscount unequal units
Key Problems with Library StatisticsKey Problems with Library Statistics
• Data imprecision due toData imprecision due to
− Inconsistent collection methodsInconsistent collection methods− MistakesMistakes− Sampling errorSampling error− “ “Gaming”Gaming”
• Imprecision makes individual library Imprecision makes individual library comparisons less accuratecomparisons less accurate
Key Problems with Library StatisticsKey Problems with Library Statistics
• Variety of reasons for insufficient scores:Variety of reasons for insufficient scores:
Key Problems with Library StatisticsKey Problems with Library Statistics
- Inadequate knowledge of community needs
- Staff skill deficiencies
- Inadequate staffing
- Inefficient workflows
- Inadequate planning
- Limited user competencies
... and others
Adapted from Poll and te Boekhorst (2007)
Output measures “reflect the interaction of Output measures “reflect the interaction of
users and library resources, users and library resources, constrained by constrained by the environment in which they operate.the environment in which they operate. The The meaning of a specific score on any measure meaning of a specific score on any measure depends on a broad range of factors depends on a broad range of factors including the library’s goals, the current including the library’s goals, the current circumstances of the library and its circumstances of the library and its environment, the users, the manner in which environment, the users, the manner in which the measure was constructed, and how the the measure was constructed, and how the data were collected.” data were collected.” [emphasis added][emphasis added]
- Van House, Weill, and - Van House, Weill, and McClure (1990)McClure (1990)
Improvements NeededImprovements Needed
• Fuller understanding of limitations of Fuller understanding of limitations of statistical indicators and comparison statistical indicators and comparison methodsmethods
“ “The [input and output] measures are best The [input and output] measures are best used with other information about the used with other information about the library.”library.”
- Van House, Weill, and McClure (1990)- Van House, Weill, and McClure (1990)
• Relate amounts and types of resources Relate amounts and types of resources and services to and services to verifiable levels of verifiable levels of community needcommunity need
• Increased understanding of Increased understanding of measurement and interpretationmeasurement and interpretation
− Draw Draw reasonable reasonable conclusions and conclusions and interpretationsinterpretations
− Study behavioral science measurement Study behavioral science measurement practicespractices
Improvements NeededImprovements Needed
Behavioral Science Behavioral Science Measurement ModelMeasurement Model
Conceptualization
Nominal Definition
OperationalDefinition
Measurement in Real World
Babbie (2007)
RecommendationsRecommendations
• Promote view of public library national Promote view of public library national rating systems as ‘contests’ with simple, rating systems as ‘contests’ with simple, arbitrary rulesarbitrary rules
• Ratings are:Ratings are:
− Quite approximate indicators of Quite approximate indicators of performanceperformance
− Inadequate indicators of quality, Inadequate indicators of quality, excellence, and effectivenessexcellence, and effectiveness
RecommendationsRecommendations
• Advise libraries to interpret national Advise libraries to interpret national ratings cautiously and conscientiouslyratings cautiously and conscientiously
• Develop and test measurement Develop and test measurement instruments for identifying peer librariesinstruments for identifying peer libraries
• Use ratings to inspire more robust Use ratings to inspire more robust measurement of public library measurement of public library performanceperformance