snowball metrics: university-owned benchmarking to reveal strengths within all activities - dr. lisa...
DESCRIPTION
June 18, 2014 NISO Virtual Conference: Transforming Assessment: Alternative Metrics and Other Trends Snowball Metrics: University-owned Benchmarking to Reveal Strengths within All Activities - Dr. Lisa Colledge, Snowball Metrics Program Director, ElsevierTRANSCRIPT
Presented by Dr Lisa Colledge
Snowball Metrics Programme Director, Elsevier
18 June 2014
www.snowballmetrics.com
Snowball Metrics:University-owned Benchmarking to Reveal Strengths within All Activities
1
The origins of Snowball Metrics
Competitive award won by Imperial College London and Elsevier to investigate the state of research management in England
2
Report available at http://www.snowballmetrics.com/wp-content/uploads/research-information-management1.pdf
The origins of Snowball Metrics
Competitive award won by Imperial College London and Elsevier to investigate the state of research management in EnglandClear trends were voiced:• “Unless you have [data] you
cannot make informed decisions; you would be acting based on opinions and hearsay.”
3
Report available at http://www.snowballmetrics.com/wp-content/uploads/research-information-management1.pdf
The origins of Snowball Metrics
Competitive award won by Imperial College London and Elsevier to investigate the state of research management in EnglandClear trends were voiced:• “Unless you have [data] you
cannot make informed decisions; you would be acting based on opinions and hearsay.”
• “[There is little] thought leadership and knowledge development around best practice.”
4
Report available at http://www.snowballmetrics.com/wp-content/uploads/research-information-management1.pdf
The origins of Snowball Metrics
Competitive award won by Imperial College London and Elsevier to investigate the state of research management in EnglandClear trends were voiced:• “Unless you have [data] you
cannot make informed decisions; you would be acting based on opinions and hearsay.”
• “[There is little] thought leadership and knowledge development around best practice.”
5
Report available at http://www.snowballmetrics.com/wp-content/uploads/research-information-management1.pdf
• “The principle drivers for our systems are often external… but they shouldn’t be… a research strategy should… be developed… to respond to our strengths and the external environment, our systems should be defined to run our business.”
University recommendations
6
Universities and funders should work more
collaboratively, and develop stronger relationships with
suppliers“Universities should work together more to make their collective
voice heard by external agencies.”
“The lack of a long-term vision makes it hard to… co-operate within a university let alone
across the sector.”
“Suppliers do not know what research offices do on a daily basis.” “How educated are we at asking suppliers the right
questions?”
1
University recommendations
7
Universities and funders should work more
collaboratively, and develop stronger relationships with
suppliers“Universities should work together more to make their collective
voice heard by external agencies.”
“The lack of a long-term vision makes it hard to… co-operate within a university let alone
across the sector.”
“Suppliers do not know what research offices do on a daily basis.” “How educated are we at asking suppliers the right
questions?” “Someone needs to take ownership of the process: it is impossible to please all of the people all of the
time so somebody needs to be strong enough to stand behind decisions and follow through.”
“It would be great if the top five
[universities] could collaborate.”
1
2
Snowball Metrics are…
• Endorsed by a group of distinguished UK universities to support their strategic decision making
• Tried and tested methodologies that are available free-of-charge to the higher education sector
• Absolutely clear, unambiguous definitions enable apples-to-apples comparisons so universities can benchmark themselves against their peers to judge the excellence of their performance
8
Snowball Metrics are…
• Endorsed by a group of distinguished UK universities to support their strategic decision making
• Tried and tested methodologies that are available free-of-charge to the higher education sector
• Absolutely clear, unambiguous definitions enable apples-to-apples comparisons so universities can benchmark themselves against their peers to judge the excellence of their performance
Snowball Metrics are unique because:• Universities drive this bottom
up• Academia – industry
collaboration
9
Snowball Metrics approach
10
Vision: Snowball Metrics enable benchmarking by driving quality and efficiency across higher education’s research and enterprise activities, regardless of system and supplier• Bottom-up initiative: universities define and
endorse metrics to generate a strategic dashboard. The community is their guardian
• Draw on all data: university, commercial and public
• Ensure that the metrics are system- and tool-agnostic
• Build on existing definitions and standards where possible and sensible
The output of Snowball Metrics
11
www.snowballmetrics.com/metrics
“Recipes” – agreed and tested metric methodologies – are the output of Snowball Metrics
From Statement of Intent:• Agreed and tested
methodologies… are and will continue to be shared free-of-charge
• None of the project partners will at any stage apply any charges for the methodologies
• Any organization can use these methodologies for their own purposes, public service or commercial
Statement of Intent available at http://www.snowballmetrics.com/wp-content/uploads/Snowball-Metrics-Letter-of-Intent.pdf
Founding project partners
12
Snowball Metrics Project Partners
Main roles and responsibilities
• Everyone covers their own costs• Universities
– Agree the metrics to be endorsed as Snowball Metrics
– Determine methodologies to generate the metrics in a commonly understood manner to enable benchmarking, regardless of systems
• Elsevier– Ensures that the methodologies are feasible– Distribute the outputs using global communications
networks– Day-to-day project management of the global
program
• Outside the remit of the Snowball Metrics program– Nature and quality of data sources used to generate
Snowball Metrics – Provision of tools to enable generation and use
Snowball Metrics
13
Snowball Metrics Vision
Snowball Metrics drive quality and efficiency across higher education’s research and
enterprise activities, regardless of system and supplier
Key aims• Agree, test and share metrics recipes in all
areas of the “research and enterprise landscape”
• Drive adoption of Snowball Metrics as global standards for institutional benchmarking– By implementation in tools, systems and reports– By engaging with university groups in key
research-intensive nations14
The Snowball Metrics Landscape
15
Denominators
16
Recipes available
17
Altmetrics
Snowball Metrics • needs to be able to aggregate counts beyond article level• Needs to be able to share the entire recipe
Altmetrics
Scholarly ActivityNumber of times that an institution’s output has been posted in online tools that are typically used by academic scholars e.g. Mendeley, CiteULike, Academia.edu, ResearchGate
Scholarly CommentaryNumber of times that an institution’s output has been commented on in online tools that are typically used by academic scholars e.g. science blogs, video posts, peer reviews, post-publication comments, Wikipedia
Mass MediaNumber of times that an institution’s output has been referred to by press clippings and news websites
Social ActivityNumber of times that an institution’s output has stimulated social media posts e.g. Facebook, Twitter, Reddit, LinkedIn
Snowball Metrics • needs to be able to aggregate counts beyond article level• Needs to be able to share the entire recipe
Altmetrics
Scholarly ActivityNumber of times that an institution’s output has been posted in online tools that are typically used by academic scholars e.g. Mendeley, CiteULike, Academia.edu, ResearchGate
Scholarly CommentaryNumber of times that an institution’s output has been commented on in online tools that are typically used by academic scholars e.g. science blogs, video posts, peer reviews, post-publication comments, Wikipedia
Mass MediaNumber of times that an institution’s output has been referred to by press clippings and news websites
Social ActivityNumber of times that an institution’s output has stimulated social media posts e.g. Facebook, Twitter, Reddit, LinkedIn
Snowball Metrics • needs to be able to aggregate counts beyond article level• Needs to be able to share the entire recipe
Scholarly Activity Scholarly Activity per FTE Scholarly Activity per publication
Globalizing Snowball Metrics
21
US• University of Michigan• University of Minnesota• Northwestern University• University of Illinois at
Urbana-Champaign • Arizona State University • MD Anderson Cancer
Center• Kansas State University
Australia / New Zealand• University of Queensland• University of Western Australia • University of Auckland • University of Wollongong • University of Tasmania• Massey University • The University of Canberra• Charles Darwin University
Interest and support from:• Japan RU11 metrics group• Association of Pacific Rim Universities (APRU)• European Commission for H2020• Fundação para a Ciência e a Tecnologia (FCT) in
Portugal
Supporting Snowball Metrics
• Feedback about how useful it would be to you to have comparative information based on the Snowball Metrics
• Feedback on the clarity of the recipes in this book
• Call for influential bodies in the social sciences and arts and humanities to collaborate with the initiative to agree on practical metrics that are particularly useful for these areas
• Ask funders and suppliers to adopt Snowball Metrics. Change requires momentum: please help us provide it
THANK YOU FORYOUR TIME AND
ATTENTION!