Snowball Metrics

Slides from :

Anna Clements, University of St Andrews

Lisa Colledge, Elsevier

Stephen Conway, University of Oxford

Keith Jeffery, euroCRIS

04/20/23Snowball Metrics Project PartnersUniversity of Oxford, University College London, University of Cambridge, Imperial Colledge London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier

1

Snowball Metrics are…

• Tried and tested methodologies that are available free-of-charge to the higher education sector

• Defined and agreed by Higher Education Institutions to support their strategic decision making

• Absolutely clear definitions enable apples-to-apples comparisons

• www.snowballmetrics.com

04/20/23 2Snowball Metrics Project PartnersUniversity of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier

The origins of Snowball Metrics

Highlighted recommendations from the study

•Institutions and funders should work more collaboratively, and develop stronger relationships with suppliers

•An agreed national framework for data and metric standards is needed

•Suppliers should participate in the development of data and metric standards

Sentiments among English research institutions were the prelude to this partnership

•Growing recognition of the value of data/metrics to inform and monitor institutions’ research strategies

•Dissatisfaction with available tools: multiple custom implementations, incompatibility of data systems

•Frustration over the lack of standard metrics available to give sensible, relevant measurements

•Frequent data requests from external bodies put pressure on institutions to collect and return data on aspects of performance that are not necessarily best suited to the institutions themselves

Imperial College London and Elsevier conducted a study of research information management in England funded by the Joint Information Systems Committee (JISC)

... which led to Snowball MetricsSnowball Metrics Project PartnersUniversity of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier

Defining Snowball Metrics

A subset of participants initiated a second phase

of work that would address and build on these

recommendations

• Vision: Institutions agree on metrics and methodologies as standards to support strategic decision making by enabling benchmarking across institutions globally

• Approach: Eight UK higher educations institutions working with a supplier, Elsevier, partnered to reach consensus on a set of metrics. Snowball Metrics reflect universities’ view of the world, not Elsevier’s. Elsevier played a supporting role to help higher education institutions articulate their needs, and to test the feasibility of addressing them

• Outcome: Agreed and tested Snowball Metrics methodologies have been shared freely in the Recipe Book, so that any organization can adopt the framework

• Next steps: All project partners are committed to facilitating a global definition of these metrics as standards by incorporating other existing standards into the “recipes”, and working with all suppliers to implement these metrics

Project Partners

Snowball Metrics Project PartnersUniversity of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier

Snowball Metrics Landscape

Snowball Metrics Project PartnersUniversity of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier

• Metrics are sliced at the institutional and discipline / funder type levels

• Metrics need a combination of institutional, proprietary, and public data

Snowball Metrics shared in Recipe Book

04/20/23 6Snowball Metrics Project PartnersUniversity of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier

Input MetricsInput Metrics

Scholarly OutputCitation Counth-indexField-Weighted Citation ImpactPublications in Top PercentilesCollaboration

Process MetricsProcess Metrics

Output MetricsOutput Metrics

Applications VolumeAwards Volume

Income VolumeMarket Share

The value of agreeing definitions• Certainty that we are comparing like with like• Identify common framework for comparisons

– Institutional organizational structures are different and not suitable to provide context

– Agreed a common framework in the UK to benchmark within: HESA Cost Centres

– We needed to assign our data to the HESA Cost Centre framework

04/20/23 7

HESA Cost Centre: a grouping of researchers by field that allows meaningful comparisons between different types of data within a

discipline

HESA: Higher Education Statistics Agency www.hesa.ac.uk

Snowball Metrics Project PartnersUniversity of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier

Specifying Awards Volume

04/20/23 8

Removing ambiguity (extracts from the complete “Recipe”)

http://www.snowballmetrics.com/metrics/Snowball Metrics Project Partners

University of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier *Ordered according to productivity 2011 (data source: Scopus)

• Use aggregated values of awards over award lifetime, not the value (to be) spent in any financial year

• Date used is the date that the award is entered in the institutional grants system

• Include subsequent financial amendments – supplements and reductions

• Do not include non-financial amendments such as no-cost extensions

Metrics are tested for feasibility

04/20/23Snowball Metrics Project PartnersUniversity of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier

9

Metrics are tested for feasibility

04/20/23Snowball Metrics Project PartnersUniversity of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier

10

Metrics are tested for feasibility

04/20/23 11Snowball Metrics Project PartnersUniversity of Oxford, University College London, University of Cambridge, Imperial College London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier *Ordered according to productivity 2011 (data source: Scopus)

Our task – to CERIFy the metrics

04/20/23Snowball Metrics Project PartnersUniversity of Oxford, University College London, University of Cambridge, Imperial Colledge London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier

12

MICE Project: CERIF Architecture

CERIF Indicators Segment

Physical data Structure Indicators

04/20/23Snowball Metrics Project PartnersUniversity of Oxford, University College London, University of Cambridge, Imperial Colledge London, University of Bristol, University of Leeds, Queen’s University Belfast, University of St. Andrews, Elsevier *Ordered according to productivity 2011 (data source: Scopus)

16