SCHOLARLY IMPACT

METRICSAN OVERVIEW

JOHAN BOLLEN – JBOLLEN@INDIANA.EDU

INDIANA UNIVERSITY

SCHOOL OF INFORMATICS AND COMPUTING

CENTER FOR COMPLEX NETWORKS AND SYSTEMS RESEARCH

OAI8 - June 2013

OAI8 - June 2013

OAI8 - June 2013

SCIENCE: IDEAS NOT BRICKSScience and scholarly communication matters.

1) Economic and cultural value is enormous, and rests on considerable investments of

1) Capital

2) Infrastructure

3) Human resources

4) Education

2) Outcomes: ideas and information

1) Not the amount of paper pulp produced, number of bricks laid, metal forged, tractors built, fields plowed

2) It’s largely about the ideas and how they are communicated, BUT:

1) Not all ideas matter equally

2) Not all ideas should be communicated

OAI8 - June 2013

SCIENCE AS A GIFT ECONOMY

Gift economy:

- services and good are shared freely without implicit or explicit expectation/agreement of reciprocation

- “economy of abundance, not scarcity”

- found in some societies

Science is a little like that:

- information is shared as freely as possible through publications

- information is perishable (half-life of good idea)

- reward for sharing is essentially a social phenomenon: “esteem”, “prestige”, “influence”

OAI8 - June 2013

IMPACT ~ PUBLICATION

Scholarly outcomes and ideas are traditionally perceived to be mainly shared through the peer reviewed literature, aka publications

- An entire industry has emerged to support this modus operandi

- Not universal, has not always been that way, might not always be this way, but presently dominant

Our ideas of scholarly impact is now strongly tied to scholarly publications

- Ideas that impact or influence fellow scholars reach them via peer-reviewed publications

- Influence and impact is thus expected to be expressed through the medium of peer-reviewed publications

-> Citation data has become de facto currency of impact or influence:

• When one scholar cites the work of another, this is deemed recognition of their influence

• Measuring impact from citations

OAI8 - June 2013

CITATION DATA

OAI8 - June 2013

CITATION NETWORKS

Maps of random walks on complex networks reveal community structure Martin Rosvall*,† and Carl T. Bergstrom*, PNAS 105(4), 1118-1123

The map equation M. Rosvall , D. Axelsson , and C.T. Bergstrom, European Journal of Physics, 178, 13–23 (2009)

OAI8 - June 2013

FROM CITATION DATA TO JOURNAL IMPACT FACTOR

Impact Factor = mean 2 year citation rate

20032001 2002

Journal x All (2003)

OAI8 - June 2013

THAT CONCLUDES THIS LECTURE

Thank you for your undivided attention.

OAI8 - June 2013

HOLD ON

It’s just not that simple

OAI8 - June 2013

A FEW THINGS LEFT TO DISCUSS…

OAI8 - June 2013

“THE MAP IS NOT THE TERRITORY”

Impact, influence is a social phenomenon

• It already exists in the scholarly community

• Most scholars already have a notion of which ideas, publications, journals, and authors matter the most

To measure this social construct of scholarly impact we can choose many different “operationalizations”/measurements:

• Ask scientists: surveys, questionnaires, awards

• Correlates: funding decisions, publication data, citation data

• “Behavioral” data: readership, ILL, reshelving download data, Twitter mentions, etc.

OAI8 - June 2013

MANY PERMUTATIONS

1. Data type and which community it represents

• Citation data: authors

• Usage data: authors, readers, public

• Social media data: everyone

2. Type of metric calculated from (1)

• Counts, normalized counts

• Social network metrics

• Trend metrics

3. Level of granularity:

• Entities: authors, journals, articles, teams, countries

• Time: 5-year span, 2 year span, etc.

OAI8 - June 2013

METRICS, CUBED

Data

type

Metric type

Granu

larit

y

citation

usage

Social mediaau

thor ar

ricle jo

urna

l

coun

ts

Social

net

work

trend

s

BACK TO CITATION DATA AND NETWORKS

Johan Bollen, Herbert Van de Sompel and Marko A. Rodriguez. Towards usage-based impact metrics: first results from the MESUR project, JCDL 2008, Pittsburgh, PA, June 2008. (arXiv:0804.3791v1)

OAI8 - June 2013

- Author-level metrics:

- Total citations- H-index:

- Nth publication with at least n citations (rank order pubs by decr. Cites)

- g-index, e-index, a-index- Co-author network indicators

- Article level metrics:

- Total citations- Normalized citation counts

- Journal level:

- Impact factor- SNIP, Crown indicator- Social network metrics from citation

network (next slide: PageRank, Eigenfactor, Y-factor, betweenness, etc)

CITATION-BASED METRICS

Radicchi et al . (2008) PNAS 105(45) 17268-17272

Hirsch (2005) PNAS 102(46) 16569-16572

Degree• In-degree• Out-degree

Shortest path• Closeness• Betweenness

Random walk• PageRank• Eigenvector

INNOVATION I : CITATION-BASED SOCIAL NETWORK METRICS

OAI8 - June 2013

SOCIAL NETWORK ANALYSIS

PAGERANK FOR JOURNALS

2003 JCR, Science Edition5709 journals, L=0.85

Pinski, G., & Narin, F. (1976). Citation influence for journal aggregates of scientific publications: theory, with application to the literature of physics. Information processing and management, 12(5), 297-312.Chen, P., Xie, H., Maslov, S., & Redner, S. (2007). Finding scientific gems with Google. Journal of Informetrics, 1(1), arxiv.org/abs/physics/0604130.

POPULARITY VS. PRESTIGEOutliers reveal differences in aspects of “status”

IF ~ general popularityPR ~ prestige, influence

Johan Bollen, Marko A. Rodriguez, and Herbert Van de Sompel. Journal status. Scientometrics, 69(3), December 2006 (DOI: 10.1007/s11192-006-0176-z)

Philip Ball. Prestige is factored into journal ratings. Nature 439, 770-771, February 2006 (doi:10.1038/439770a)

PAGERANK FOR JOURNALS

OAI8 - June 2013

Scholarly community and communication is moving online.

Data pertaining to online activities (implicit, behavioral) vs. citation data (explicit declaration of influence)

INNOVATION II: “BEHAVIORAL” DATA

Scholarly community

Scholarlycommunication

items

metrics

Behavioral data

Bibliographic data

Citation

OAI8 - June 2013

BEHAVIORAL DATA

Reading/usage statistics

• Interlibrary loan data

• Reshelving data

• Online catalogue systems

Daily, weekly, monthly access or reading statistics

Usage data:

• Web server logs

• Link resolver data (SFX, etc)

Detailed data on “who”, “what”, “where”, “when”: ability to track scholarly activity in real-time

OAI8 - June 2013

USAGE STATISTICS

COUNTER: member organization defining an auditable standard for reporting and aggregating monthly usage statistics (www.projectcounter.org)

• Journal and article level

• Initiative to define “usage factor”

PLoS Article Level Metrics

• Download numbers

• download trends

OAI8 - June 2013

MESURAndrew W. Mellon and NSF funded project at LANL Digital Library Research and Prototyping and Indiana University

- Very large-scale usage data from publishers, aggregators, and library consortia

- Metrics of scholarly impact derived from aggregated usage data

- Mapping scientific activity from log clickstream data

- Examine“scholarly impact” itself (more later!)

Notable distinction: use of log data that contains clickstream enables metrics and analysis beyond level of usage statistics

Presently concluding planning process (Andrew W. Mellon funded) to evolve to community-supported, sustainable entity

OAI8 - June 2013

INNOVATION III: ALT-METRICSBehavioral AND “attention” data.

• Social media attention, bookmarking, mentions

• Attempt to also capture “social” attention or public impact of scholarly work (not just articles!), another possible dimension of impact

OAI8 - June 2013

SOME RELEVANT RESEARCH

Shuai X, Pepe A, Bollen J (2012) How the Scientific Community Reacts to Newly Submitted Preprints: Article Downloads, Twitter Mentions, and Citations. PLoS ONE 7(11): e47523. doi:10.1371/journal.pone.0047523

Eysenbach G (2011) Can tweets predict citations? Metrics of social impact based on twitter and correlation with traditional metrics of scientific impact. Journal of Medical Internet Research 13: e123.

OAI8 - June 2013

TWITTER MENTIONS ~ DOWNLOADS, CITATIONS?

OAI8 - June 2013

TWITTER MENTIONS CORRELATE WITH DOWNLOADS AND CITATIONS!

OAI8 - June 2013

ALT-METRICS AS PART OF IMPACT ASSESSMENT

OAI8 - June 2013

CITATION DATA, METRICS, IMPACT, ALT-METRICS, USAGE DATA, LET’S STEP BACK FOR A SECOND

OAI8 - June 2013

BLIND MAP-MAKERS

Odd, nearly tautological situation:

- We have many different metrics or ways to measure impact.

- But no formal or consistent definition of scholarly impact.

- No idea of what exactly impact is, how it manifests itself, what its structure is, along which dimensions it varies. etc

- Whether our metrics actually measure or represent impact

- Our metrics ARE the definition of “impact”

OAI8 - June 2013

SCHOLARLY IMPACT

Metric 4

Metric 3

Metric 2

Metric 5

Metric 6

Metric 1

impact

Not quiteimpact

Some form of impact

Not impact

Validity & Reliability

OAI8 - June 2013

MAPPING OUT IMPACT, ONE METRIC AT A TIME

• Bollen J, Van de Sompel H, Hagberg A, Chute R (2009) A Principal Component Analysis of 39 Scientific Impact Measures. PLoS ONE 4(6): e6022. doi:10.1371/journal.pone.0006022

• Priem at al. Altmetrics in the wild.

• Thelwall M, Haustein S, Larivière V, Sugimoto CR (2013) Do Altmetrics Work? Twitter and Ten Other Social Web Services. PLoS ONE 8(5): e64841. doi:10.1371/journal.pone.0064841

• PLoS ONE alt-metrics correlations: investigated by L Juhl Jensen, Novo Nordisk Foundation

• Bornmann, L., Mutz, R., & Daniel, H.-D. (2008). …A comparison of nine different variants of the h index using data from biomedicine. JASIST, 59(5), 830-837.

OAI8 - June 2013

FINALLY… WHY?Just like social status, scholarly impact (or other) is an interesting scientific study area. It emerges from the scholarly communication process.

BUT pure science is clearly not the only motivation:

• Metrics used in assessment

• Decision-making: funding, promotion, …

• Information filtering

Some of these applications are tremendously useful and potentially enabling of radical changes in scholarly communication, e.g. information filtering and assessing broader community impact of scholarly work.

OAI8 - June 2013

HOWEVER…

Assuming that scholarly impact exists, independently of whether we measure it or not:

Why measure it at all in cases where the scholarly community truly has decision-making power, autonomy? Isn’t the latter a more desirable option than administrators, politicians, and bureaucrats making decisions on the basis of numbers they don’t understand?

So buy me a beer and ask me about our crazy crowd-sourced funding idea…

Johan Bollen, David Crandall, Damion Junk, Ying Ding, Katy Boerner. Collective allocation of science funding: from funding agencies to scientific agency. http://arxiv.org/abs/1304.1067

OAI8 - June 2013

THANK YOU