research impact beyond metrics
DESCRIPTION
Traditional metrics, such as the h-index and journal impact factors, are used to measure the scholarly impact of research. However, in the current climate of accountability by funding providers, fund recipients would benefit from a more comprehensive impact management system (IMS) to facilitate the capture and reporting of narratives (including metrics) about research impact in the academy, on social policy, in industry, and ultimately with the public. Librarians have always been good at telling and facilitating stories. Research support librarians can use their storytelling skills to contribute to the implementation and administration of an impact management system. Being able to translate research impact into harvestable and reportable metadata is the key.TRANSCRIPT
![Page 1: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/1.jpg)
Impact beyond metrics: Telling your research impact story
Research Support Community DayBrisbane, Australia, 11 February 2013
Pat LoriaResearch LibrarianUniversity of Southern Queensland
![Page 2: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/2.jpg)
Limitations of traditional metrics
Scholarly Public
Recommended Faculty of 1000 Popular press
Cited Traditional citations Wikipedia
Discussed Scholarly blogs Facebook, Twitter
Saved Mendeley, CiteULike Delicious
Viewed PDF views HTML views
Adapted from Jason Priem, Altmetrics and Revolutions: https://docs.google.com/presentation/pub?id=1Y4JnchsmHHiOQdJsEpQr33qmMWqhZJrPTDAg1cZoCcI&start=false&loop=false&delayms=3000#slide=id.i0
![Page 3: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/3.jpg)
Limitations of traditional metrics
Scholarly Public
Recommended Faculty of 1000 Popular press
Cited Traditional citations Wikipedia
Discussed Scholarly blogs Facebook, Twitter
Saved Mendeley, CiteULike Delicious
Viewed PDF views HTML views
Adapted from Jason Priem, Altmetrics and Revolutions: https://docs.google.com/presentation/pub?id=1Y4JnchsmHHiOQdJsEpQr33qmMWqhZJrPTDAg1cZoCcI&start=false&loop=false&delayms=3000#slide=id.i0
![Page 4: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/4.jpg)
Three generations of impact metrics
Journal-level
metrics
Author-level
metrics
Article-level
metrics
![Page 5: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/5.jpg)
Different levels of metrics
Journal-level metrics Journal impact factor SJR & SNIP
Author-level metrics Citations per Paper (CPP) H-index
Article-level metrics Articles, datasets, blogs, code, artistic creations Citations, mentions, views, downloads, etc. Altmetric bookmarklet, ImpactStory, etc.
![Page 6: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/6.jpg)
USQ and the NCP calculator
NCP: Normalized Citations per Paper Developed by USQ (Library and Systems) Uses Scopus RDCP data to normalize Normalization creates a level playing field Citations in low visibility/low citing fields
normalized up Citations in high visibility/high citing fields
normalized down Source code for NCP calculator to be released
soon on GitHub
![Page 7: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/7.jpg)
NCP calculator
Citations: 2010; Papers: 2007-2009
![Page 8: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/8.jpg)
Measuring impact beyond academia
…impact is defined in a similar way as for the UK REF, i.e. “an effect on, change, benefit to the economy, society, culture, public policy or services, health, the environment or quality of life beyond academia”.
ATN and Go8, Guidelines for completion of case studies in ATN/Go8 EIA Impact Assessment Trial: June – August 2012, http://www.atn.edu.au/eia/Docs/EIA_Trial_Guidelines_FINAL.pdf
![Page 9: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/9.jpg)
Measuring impact beyond academia
Impact Case Study (ATN & Go8, similar to REF) Institution SEO codes Title of Case Study Context Summary of Case Study Impact Details of Impact Research underpinning Impact Research outputs from research underpinning impact Additional information:
Validation of impact People Investment income
http://www.atn.edu.au/eia/Docs/EIA_Trial_Guidelines_FINAL.pdf
• Overheads?• Data sources?
![Page 10: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/10.jpg)
Alternative metrics or altmetrics
[A]ltmetrics is the creation and study of new metrics based on the Social Web for analyzing, and informing scholarship.
http://altmetrics.org/about/
![Page 11: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/11.jpg)
Altmetric bookmarklet
http://www.altmetric.com/bookmarklet.php
![Page 12: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/12.jpg)
Altmetric bookmarklet
Free bookmark for Chrome, Firefox and Safari Click on while viewing paper for impact data Altmetric API can be embedded into second
and third-party platforms, apps and mashups Only works on PubMed, arXiv or DOI Only supports publishers who embed Google
Scholar friendly citation metadata Twitter mentions are only available for articles
published since July 2011
Example: Oxidants, antioxidants and the current incurability of metastatic cancers
![Page 13: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/13.jpg)
Altmetric bookmarklet
Counts Disadvantages / Bugs
Altmetric score Difficult to explain
Tweeters
Facebook users
News outlets
Science blogs
Google+ users
Mendeley readers
CiteULike readers
Connotea readers
Bookmarklet only works on PubMed, arXiv or pages containing a DOI
Only supports publishers who embed Scholar friendly citation metadata
Twitter mentions are only available for articles published since July 2011
http://www.altmetric.com/bookmarklet.php
![Page 14: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/14.jpg)
ImpactStory.orgArtefact type Counts Known Bugs 12/12
Journal article Topsy Twitter coverage weak
Dataset Science Seeker Different results for DOI and URL searchesSoftware PubMed
Slides Dryad Missing metrics
Generic SlideShare Metrics too low
Wikipedia
Figshare
PLoS Search
PLoS ALM
Delicious
Facebook No link to mentions
Scopus
CiteULike
GitHub
Mendeley
![Page 15: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/15.jpg)
ImpactStory.org
![Page 16: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/16.jpg)
![Page 17: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/17.jpg)
![Page 18: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/18.jpg)
PlumAnalytics.com
Usage - Downloads, views, book holdings, ILL, document delivery
Captures - Favourites, bookmarks, saves, readers, groups, watchers
Mentions - blog posts, news stories, Wikipedia articles, comments, reviews
Social media - Tweets, +1's, likes, shares, ratings
Citations - Web of Science, Scopus, Google Scholar, Microsoft Academic Search
![Page 19: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/19.jpg)
PlumAnalytics.com
articles book chapters books clinical trials datasets figures grants patents
presentations source code videos greatest sources researcher graph group metrics paid service no API available
![Page 20: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/20.jpg)
Reasons to use altmetrics
Same reasons as using traditional metrics: Grants, promotions, staff/program review Review research dissemination strategy
Measure of influence and reach of output Publishers can add value for authors/readers Informed decisions by research managers Comprehensive view of impact for funders “Citation graph data is like Chekhov’s gun:
once on stage, it has to be fired.” (Peter Vinkler, cited by Jason Priem, Altmetrics and Revolutions)
![Page 21: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/21.jpg)
Article Level Metrics (ALMs)
“Article-Level Metrics are a comprehensive set of impact indicators that enable numerous ways to assess and navigate research most relevant to the field” (http://article-level-metrics.plos.org/alm-info/)
![Page 22: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/22.jpg)
PLoS ALMs
http://article-level-metrics.plos.org/alm-info/
![Page 23: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/23.jpg)
PLoS example
![Page 24: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/24.jpg)
![Page 25: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/25.jpg)
Source: http://article-level-metrics.plos.org/researchers/
![Page 26: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/26.jpg)
Research profile systems
For example, Symplectic Elements:
![Page 27: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/27.jpg)
The great objection!
It’s not real impact!= It’s not scholarly impact.
Research Impact Story
Scholarly Public
Government Industry
![Page 28: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/28.jpg)
What is USQ’s research impact story?
![Page 29: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/29.jpg)
![Page 30: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/30.jpg)
Impact management system (IMS)
IMS
Impact Metrics
Curriculum Vitae
Esteem Measures
Professional Activities
Industry Impact
Policy Impact
Community Impact
Research Income
Human Resources
Research Outputs
Flexible Reportable
Metadata is key!
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
• Metadata• Metadata• Metadata
![Page 31: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/31.jpg)
Impact metadata
Academic Citation Publication Teaching Supervision Research project Research income
Government Application Consultation Social policy
Industry Consultation Partnership Patent
Social Altmetric
Facebook Twitter Mendeley Wikipedia
Engagement Media coverage
Design according to reporting needs
![Page 32: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/32.jpg)
Discussions with Symplectic
![Page 33: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/33.jpg)
Librarian as storyteller
Image: http://kassonpubliclibrary.blogspot.com.au/2010/03/spring-planting-at-library-storytime.html
![Page 34: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/34.jpg)
Advantages of IMS
Facilitates storytelling of research impact Saves time for researchers and managers Enables internal and external reporting Evidence base for individual/program review Accommodates government requirements Data for case studies and grant applications Impact captured for longitudinal analysis Best IMS support transfer of data to other IMS Managing impact: a new research literacy?
![Page 35: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/35.jpg)
Take-aways
Holistic impact monitoring and reporting Make it easy for academics and managers Do you need an impact management system? What is your research impact story?
http://jeps.efpsa.org/blog/2012/06/20/maximizing-research-impact/
![Page 36: Research impact beyond metrics](https://reader038.vdocuments.site/reader038/viewer/2022103015/54c633bb4a7959c9458b4577/html5/thumbnails/36.jpg)
Research impact discussion
Image: http://www.duffysrehab.com/blog/want-to-make-an-impact-consider-betty-ford
Pat LoriaEmail: [email protected]: @pat_loria