alternative assessment metrics initiateve meeting 1 opening
DESCRIPTION
Todd Carpenter's presentation at the opening of the Alternative Assessment discussion forum in San Francisco, CA on 10/09/13TRANSCRIPT
![Page 1: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/1.jpg)
Do We Need Standards for
Alternative Metrics?
What are we hoping to accomplish?
Todd Carpenter, Executive Director, NISONISO Alternative Assessment Initiative
Discussion Forum #1 San Francisco, CA | October 9, 2013
Friday, October 11, 13
![Page 2: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/2.jpg)
• Non-profit industry trade association accredited by ANSI with 150+ members
• Mission of developing and maintaining technical standards related to information, documentation, discovery and distribution of published materials and media
• Volunteer driven organization: 400+ spread out across the world
• Responsible in various wasy for standards like ISSN, DOI, Dublin Core metadata, DAISY digital talking books, OpenURL, MARC records, and ISBN
About
Friday, October 11, 13
![Page 3: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/3.jpg)
We stand on the shoulders of giants...
Friday, October 11, 13
![Page 4: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/4.jpg)
The Mighty Citation
Friday, October 11, 13
![Page 5: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/5.jpg)
Impact factor
Friday, October 11, 13
![Page 6: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/6.jpg)
Thomson-Reuters IMPACT FACTOR
Friday, October 11, 13
![Page 7: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/7.jpg)
Two Stanford Students
Friday, October 11, 13
![Page 8: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/8.jpg)
They wrote a paper
Friday, October 11, 13
![Page 9: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/9.jpg)
They started a company in someone’s garage
Photo: APFriday, October 11, 13
![Page 10: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/10.jpg)
Friday, October 11, 13
![Page 11: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/11.jpg)
Citations are awesome, aren’t they?
A perfect solution to all the world’s problems, no?
A simple conclusion can be drawn
Friday, October 11, 13
![Page 12: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/12.jpg)
“What Des-Cartes did was a good step. You have added much several ways, & especially in taking ye colours of thin plates into philosophical consideration. If I have seen further it is by standing on ye shoulders of Giants."
Back to Sir Isaac Newton
Friday, October 11, 13
![Page 13: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/13.jpg)
Citations aren’t always what we think they are
Caveat emptor
Friday, October 11, 13
![Page 14: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/14.jpg)
Another problem with citations
Friday, October 11, 13
![Page 15: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/15.jpg)
So where to from here?Friday, October 11, 13
![Page 16: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/16.jpg)
Define altmetrics, please
Friday, October 11, 13
![Page 17: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/17.jpg)
Usage Based Metrics
Friday, October 11, 13
![Page 18: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/18.jpg)
Clickstream Analysis
Friday, October 11, 13
![Page 19: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/19.jpg)
Social Media Metrics
Image: Danny Brown, The State of Social Media Marketing 2012
Friday, October 11, 13
![Page 20: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/20.jpg)
Clickstream data yields high-resolution maps of science
Johan Bollen, Herbert Van de Sompel, et al. PLoS One, February 2009
Clickstream network analysis
Friday, October 11, 13
![Page 21: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/21.jpg)
Clickstream data yields high-resolution maps of science
Johan Bollen, Herbert Van de Sompel, et al. PLoS One, February 2009
Clickstream network analysis
Friday, October 11, 13
![Page 22: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/22.jpg)
Behavioral Metrics
Friday, October 11, 13
![Page 23: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/23.jpg)
Multivariate analysis
Friday, October 11, 13
![Page 24: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/24.jpg)
Sentiment Analysis
Friday, October 11, 13
![Page 25: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/25.jpg)
Image: Domenico, Caron, Davis, et al.
Citation of non-traditional content
Citations to new forms of communication aren’t as simple as one might think.
Define a use of this?
Friday, October 11, 13
![Page 26: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/26.jpg)
We’ve got what, now how?
Friday, October 11, 13
![Page 27: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/27.jpg)
Image: Flickr user karindalziel
What can be measured?
Friday, October 11, 13
![Page 28: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/28.jpg)
How do we measure things?
Friday, October 11, 13
![Page 29: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/29.jpg)
At what granularity do we measure things?
Friday, October 11, 13
![Page 30: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/30.jpg)
Which measurements are valuable?
Friday, October 11, 13
![Page 31: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/31.jpg)
How do we ensure the validity of those measurements?
Friday, October 11, 13
![Page 32: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/32.jpg)
How to (or even if we should) deal with other issues
such as gaming?
Friday, October 11, 13
![Page 33: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/33.jpg)
Issues we face with alternative
metrics
Friday, October 11, 13
![Page 34: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/34.jpg)
Basic Definitions
(So we are all talking about the same thing)
Friday, October 11, 13
![Page 35: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/35.jpg)
Comparison across providers
Source: Scott Chamberlain, Consuming Article-‐Level Metrics:Observations And Lessons From Comparing Aggregator Provider Data, Information Standards Quarterly, Summer 2013, Vol 25, Issue 2.
Friday, October 11, 13
![Page 36: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/36.jpg)
Consistency in what is counted
Source: Scott Chamberlain, Consuming Article-‐Level Metrics:Observations And Lessons From Comparing Aggregator Provider Data, Information Standards Quarterly, Summer 2013, Vol 25, Issue 2.
Friday, October 11, 13
![Page 37: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/37.jpg)
Authorship-----
Disambiguation&
Contributor Roles
Friday, October 11, 13
![Page 38: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/38.jpg)
Open exchange of component data
Friday, October 11, 13
![Page 39: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/39.jpg)
TRUST
Friday, October 11, 13
![Page 40: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/40.jpg)
What is NISO trying to do?Friday, October 11, 13
![Page 41: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/41.jpg)
Friday, October 11, 13
![Page 42: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/42.jpg)
Friday, October 11, 13
![Page 43: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/43.jpg)
Alterna(ve Assessment Ini(a(ve
Phase 1 Mee(ngsOctober 9, 2013 -‐ San Francisco, CADecember 11, 2013 -‐ Washington, DC
January 23-‐24 -‐ Philadelphia, PA
Phase 1 report expected in May 2014
Friday, October 11, 13
![Page 44: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/44.jpg)
Alterna(ve Assessment Ini(a(ve Phase 2
Presenta(ons of report (June 2014)Priori(za(on Effort (June -‐ Aug, 2014)
Project approval (Sept 2014)Working group forma(on (Oct 2014)
Consensus Development (Nov 2014 -‐ Dec 2015)Trial Use Period (Dec 15 -‐ Mar 16)
Publica(on of final recommenda(ons (Jun 16) Friday, October 11, 13
![Page 45: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/45.jpg)
Criticism
Friday, October 11, 13
![Page 46: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/46.jpg)
Why now?Isn’t it too soon?
Can NISO to do this?
Friday, October 11, 13
![Page 47: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/47.jpg)
How soon is now?
Friday, October 11, 13
![Page 48: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/48.jpg)
These things take time
Friday, October 11, 13
![Page 49: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/49.jpg)
It will be hard to changeFriday, October 11, 13
![Page 50: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/50.jpg)
Friday, October 11, 13
![Page 51: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/51.jpg)
“All I see now is blonde, brunette,
redhead”
Friday, October 11, 13
![Page 52: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/52.jpg)
One Very Last Thing…
Image from Will Lion: http://www.flickr.com/photos/will-lion/2646213692/
Friday, October 11, 13
![Page 53: Alternative assessment metrics initiateve meeting 1 opening](https://reader035.vdocuments.site/reader035/viewer/2022062319/555193e4b4c90580128b55f8/html5/thumbnails/53.jpg)
Thank you!
Todd Carpenter, Executive [email protected]
National Information Standards Organization (NISO)3600 Clipper Mill Road, Suite 302
Baltimore, MD 21211 USA+1 (301) 654-2512
www.niso.org
Friday, October 11, 13