beyond the factor: talking about research impact
TRANSCRIPT
Beyond the Factor:Talking about research impact
Claire StewartAssociate University Librarian for Research & Learning
Senate Library Committee, February 10, 2016
Interest in metrics
• In hiring, in support of promotion & tenure• Funders and publishers, evaluating proposals• Institutional productivity• Impact on our communities
JIF: Journal Impact Factor
Source: The Thompson Reuters Impact Factor
Significant variance across disciplines:• Top ranked journal overall:
JIF = 144.800• Top ranked journal in history:
JIF = 2.615
Not based on any single author/article
Oft criticized (DORA, Leiden, HEFCE statements)
eigenfactor
Based on same citation source as the Impact Factor (Thomson’s Journal Citation Reports)
Weights journals by importance based on citation frequency, similar to Google page rank
Also calculates an Article Influence score, over the first 5 years of an article
h index
Scholar-specific:“A scientist has index h if h of his or her Np papers have at least h citations each and the other (Np – h) papers have ≤h citations each.”
Dependent on citation index source (Google scholar and Scopus might have different values)
Doesn’t really account for different citation/usage patterns between fields
Source: Hirsch, J. E. “An Index to Quantify an Individual’s Scientific Research Output.” Proceedings of the National Academy of Sciences of the United States of America 102, no. 46 (November 15, 2005): 16569–72. doi:10.1073/pnas.0507655102.
Altmetrics
• Shares and mentions in non-traditional places, on social media, etc. (twitter, FB, Mendeley, blogs, wikipedia etc.)
• Often dependent on identifiers (DOIs, PubMedIDs, arXivIDs, etc.) which can have lower penetration in arts & humanities fields
Advice to HEFCE (UK)
Framework for responsible metrics• Robustness: use the best
possible data• Humility: quantitative should
support expert assessment (e.g., peer review)
• Transparency: be able to show where data came from & let results be verified
• Diversity: account for variation by field
• Reflexivity: indicators updated as the system & effects change
20 specific recommendations to HEFCE around use of metrics
What do we want to know when we talk about impact?
• How has this [researcher’s] work advanced knowledge?
• Has this research been evaluated and by whom? • What is field shaping research?• Who are the researchers shaping my field?• What is going on in my field that’s important, or in a
field that could benefit my work? • What is the broader societal benefit of this work?
(value of higher education, research investments)
Source: Weinberg, Bruce A., Jason Owen-Smith, Rebecca F. Rosen, Lou Schwarz, Barbara McFadden Allen, Roy E. Weiss, and Julia Lane. “Science Funding and Short-Term Economic Activity.” Science 344, no. 6179 (April 4, 2014): 41–43. doi:10.1126/science.1250055.
Where was CIC federal research funding $ actually spent?
What are the other questions we will want to ask?
And what kinds of information will we need to answer these questions?
What are the other questions we will want to ask?
Who at UMN is doing research in or about countries other than the United States? Who are they collaborating with? What kind of effect has this work had?
‘Effect’ could include: articles, books and reports published, presentations offered, information about who benefited from these outputs, integration into policy development (conversations about and/or new legislation, regulation, etc.)
Why is this hard?
• Wide variety in what constitutes a valuable research output/indicator across disciplines
• Types of outputs expanding• Data about research outputs is messy partly
because it has the typical big data problems: volume, velocity, variety
• Highly distributed scholarly communication infrastructure (the data about outputs is everywhere)
Outputs/indicators vs metrics
“The observations here relate to the fact that while there is unease about the use of metrics as a mode of
‘measuring’ the excellence of research produced in the UK’s HEIs, the rich array of data presented as part of REF2014 demonstrates that the arts and humanities
sector are comfortable with deploying numbers (albeit framed as data rather than metrics) to present a case
about the excellence of their research cultures.”
Mike Thelwall, and Maria M Delgado. “Arts and Humanities Research Evaluation: No Metrics Please, Just Data.” Journal of Documentation 71, no. 4 (June 25, 2015): 817–33. doi:10.1108/JD-02-2015-0028.