Workshop, Edinburgh University

Library

23 June 2008

John MacCollEuropean Director, OCLC Programs & Research

Research Outputs in Reputation Management

Research Outputs in Reputation Management

ProgrammeProgramme

09.15-09.30 Coffee/tea

09.30-10.15 Introduction: OCLC Programs & Research; Research outputs in reputation management overview (JMacC)

10.15-11.00 Library infrastructure: development history (Library staff)

11.00-11.20 Break

11.20-11.35 Library infrastructure continued

11.35-12.30 Research administration infrastructure: development history (Research Administration & Applications staff)

12.30-13.10 Lunch break

13.10-13.50 Facilitated discussion: objectives setting

13.50-15.30 Action plan development

15.30-15.50 Break

15.50-16.15 Outcomes (discussion)

16.15-16.30 Summing up (JMacC)

OverviewOverview

• OCLC Programs & Research

• Bad penny: an explanation

• ROiRM/WRAP

• RAE

• RQF - ERA

• Other international examples

The RLG Programs Partnership The RLG Programs Partnership

• RLG Programs provides a venue and focus for collaboration, problem-solving, and the development of new standards, products, and services among research institutions

• Around 140 libraries, museums and archives across the world with

• Deep, rich research collections

• Mandate to make collections accessible

• Commitment to exploit technology

• Commitment to collaboration

• Commitment and capability to contribute to ‘commons’ (collections, expertise, infrastructure)

• We seek to provide a strategic guidance role

Programs & ResearchPrograms & Research

San Mateo, California

• 12 Program Officers

• Vice President

• Administrative support

Dublin, Ohio

• Communications Team

• Vice President

• 20 Research Scientists

• Research Assistants

• Administrative Support

Fife, Scotland

• Director, Europe

RLG Programs Partners - GeographyRLG Programs Partners - Geography

Japan1Middle East

2

Australia andNew Zealand

4

UK, Ireland & Continental Europe

29

NorthAmerica

100

Our PartnersOur Partners

American Academy in Rome American Antiquarian Society American Museum of Natural History American Philosophical Society American

University in Cairo American University of Sharjah Amon Carter Museum Arizona State University Art Institute of Chicago Athenaeum of Philadelphia

Bard Graduate Center/Bard College Bayerische Staatsbibliothek Biblioteca Nacional de España Bibliothèque nationale de France Binghamton University, State University of New York Boston Public Library Boston University Pappas Law Library Brandeis University Brigham

Young University British Library Brooklyn Museum John Carter Brown Library at Brown University California Digital Library California Historical Society Canadian Centre for Architecture Center for Jewish History Chemical Heritage Foundation Sterling and Francine Clark Art Institute

Cleveland Museum of Art Columbia University Cornell University Courtauld Institute of Art Direction des Archives de France

Duke University Emory University Fashion Institute of Technology Florida State University Folger Shakespeare Library Fordham University Leo T.

Kissam Memorial Law Library The Frick Collection and Frick Art Reference Library George Washington University Jacob Burns Law Library Getty Research Institute Solomon R. Guggenheim Museum Hagley Museum and Library Linda Hall Library of Science, Engineering & Technology Harvard

University Hebrew Union College - Jewish Institute of Religion Huntington Library, Art Collections, and Botanical Gardens Imperial College

of Science, Technology, and Medicine Indiana University BloomingtonInstitute for Advanced Study International Institute of Social History (Netherlands) Keio University Kimbell Art Museum King's College London Library Company of Philadelphia Library of Congress Library of Virginia London School of Economics Los Angeles County Law Library Metropolitan Museum of Art Minnesota Historical Society Museum of the

American West Museum of Fine Arts, Boston Museum of Fine Arts, Houston Museum of Modern Art National Archives and Records Administration

National Gallery of Art National Gallery of Canada National Library of Australia National Library of New Zealand National Library of Scotland Natural History Museum (UK) Nelson-Atkins Museum of Art Newberry Library The New School (university) New York Academy of

Medicine New York Botanical Garden New-York Historical Society New York Public Library New York State Office of Cultural Education New York

University Oregon State University Pennsylvania State University Philadelphia Museum of Art Princeton University Rice University Rockefeller

Archive Center (Rockefeller University) Royal Botanic Gardens, Kew Rutgers University School of Oriental and African Studies Smithsonian Institution

Southern Methodist University Underwood Law Library Stanford University Stony Brook University, State University of New York St. Louis Art

Museum Swarthmore College Swiss National Library Syracuse University Temple University Trinity College Dublin UCL (University College London)

University of Aberdeen University of Alberta University of Arizona University of British Columbia Asian Library University of California, Berkeley

University of California, Davis, Law Library University of California, Los Angeles University of Cambridge University of Chicago University of Edinburgh University of Florida University of GlasgowUniversity of Glasgow University of Iowa Law Library University of Leeds University of Liverpool University of

Manchester University of Maryland University of Melbourne University of Miami University of Michigan University of Minnesota, Twin Cities

University of Oxford University of Pennsylvania University of Southern California University of Sydney University of Texas at Austin University of Toronto University of Warwick University of Washington Victoria and Albert Museum Wellcome Library for the History and Understanding of Medicine H.F. du Pont Winterthur Museum Yale University Yeshiva University

The Work Agenda

•Scholarship in text aggregations•Libraries, Archives and Museums study•Research Outputs in Reputation Management

•Shared print collections•Harmonising digitisation•Data-mining for management intelligence•Collection sharing beyond libraries

•Changing metadata creation processes •Leverage vocabularies for effective discovery •Tools and services to support metadata processes

•Create new structures and service areas•Network-level ILS (INLS)•Build prototypes/software to showcase and demonstrate new service possibilities•Investigate where which data is best exposed

(Bad penny)

ROiRM (or WRAP?): Problem statementROiRM (or WRAP?): Problem statement

• ‘due to the high cost of sustaining these frameworks on the basis of peer review of significant samples of research outputs produced by large numbers of institutions, there is discussion about moving parts of this evaluation from a peer-review process to one based on metrics (e.g. citation analysis). New metrics for assessment have recently been proposed ….’

• ‘These frameworks for assessment throw into sharp relief the interaction between university research administration and information management services potentially offered by the library. ‘

• ‘Good practice has not yet emerged, however, and the models for confronting these challenges are diverse due to the novelty of practice involved in libraries working on systems design with research administration units. There is no common view of the optimum architecture for managing research outputs comprehensively across the commercially provided component (for which a library will typically have metadata only) and the Open Access component (for which a library may provide metadata with full-text links to non-local outputs, and locally-held full-text). Nor is there yet any common agreement on an optimal set of metrics for the assessment of research such that individual researchers as well as their institutions can be easily profiled nationally and internationally.’

Project: Expertise profilingProject: Expertise profiling

‘What are the ways in which universities are disclosing academic staff expertise and productivity? Do they intersect with existing library services? In some institutions, some of these services are provided by third parties. What are the advantages and disadvantages of this?’

Project: Research output administration workflow Project: Research output administration workflow

‘What types of workflow are emerging to support research administration? Typically, libraries running research repositories are providing data which is amalgamated with financial data (e.g. on research grant income) and headcount data (e.g. on numbers of research staff and students) to compose discipline-specific profiles. Are models for efficient data-merging yet in evidence? Are third-party system vendors responding to this need? Can best practice approaches be identified and promulgated? What models have emerged in the relationship between institutional repositories and the managed publications lists that are required to drive assessment input?’

Project: Network-level discovery of research outputs Project: Network-level discovery of research outputs

‘What steps are institutions taking to promote the discoverability of their research outputs on the web? In the context of providing content which is tuned to optimise discovery by third-party search engines, what is the minimum platform investment which research libraries need to make?

Models have been proposed to encourage quality thresholds for metadata in institutional repositories to be achieved via ‘club good’ cooperative metadata approaches. What is the value of a more highly regulated cooperative approach as opposed to the lesser regulated approach of producing more variable metadata and trusting to third-party search engines such as Google to perform effectively? How closely related are enhanced metadata quality standards and ‘gaming the system’ (or ‘search engine optimisation’) in the field of research outputs?’

The context for today’s discussionsThe context for today’s discussions

• National research assessment regimes

• Impact measures

• University ranking systems

• Showcase services

• New intra-institutional collaborations and efficiencies

International assessment approachesInternational assessment approaches

(Professor Sir Gareth Roberts: ‘Caution: Evaluation measures disturb the system they purport to evaluate’)

Research Excellence FrameworkResearch Excellence Framework

‘This integrated system will combine bibliometrics and other quantitative indicators with light-touch peer review within a variable geometry of assessment. Bibliometric indicators of research quality will be a key element in quality assessment wherever this is appropriate, with light-touch review of research outputs operating where it is not. In all cases we will use other indicators of quality and impact appropriate to the disciplinary area and determined after advice from expert panels.’ - HEFCE, May 2008

REF assessmentREF assessment

• Discipline-based

• Spectrum of assessment, i.e.

‘Some subjects may be assessed through bibliometrics in combination with other quantitative indicators only (where this approach is sufficiently robust to be used without the need for qualitative elements or expert review of outputs)

For subjects where bibliometrics and other quantitative indicators are partially informative, they would be used in combination with qualitative elements and possibly some expert review of outputs

For subjects where bibliometric indicators are not sufficiently mature to be informative, expert review of outputs would be used in combination with other applicable indicators and qualitative information.’

REF timetableREF timetable

• First assessment during year 2013 ‘to drive quality-related research funding from academic year 2014-15’

• For subjects where bibliometric indicators play a leading role in quality assessment, these will start to influence HEFCE funding phased in from 2011-12.

• A full exercise to produce bibliometric indicators will take place, for appropriate subjects, in 2010.

REF and bibliometricsREF and bibliometrics

• A pilot exercise in constructing bibliometric indicators is starting now.

• ‘It will inform decisions about which subjects bibliometric indicators should be used for, how such indicators can be used in creating quality profiles and the design of the process for producing them.’

• Has Edinburgh expressed interest in the pilot?

• Results due in spring 2009

REF: still to be decidedREF: still to be decided

• Universal vs selective approach to including staff and papers

• Whether papers should be accredited to the institution or the researcher (who may move around)

• Timeframe and frequency

• WoS, SCOPUS, or both?

• Prior data cleaning requirements

REF: the role of citation analysisREF: the role of citation analysis

• How to define ‘fields’ for purposes of normalisation

• How to handle self-citation

• Profiles: how to construct them; whether internal profiles can be provided; how to benchmark them internationally

Differences between RQF and RAE2008 (Gareth Roberts)Differences between RQF and RAE2008 (Gareth Roberts)

• UK process has 67 disciplinary sub-panels

• Stronger emphasis on Research Environment and Esteem indicators in UK approach

• Different descriptors for 5-point Research Quality Rating Scale

• UK provides financial support for ‘Third Stream’ activities, i.e. provision of infrastructure for knowledge transfer (approx 10% of the funds distributed)

• RQF grades both Research Quality and its Impact (latter against a 3-point scale: high; moderate; limited)

• RQF has broader Research Impact outputs. Reports to Government and industry. Greater media and social impact.

RQF - ERA: Consultation Paper, June 2008RQF - ERA: Consultation Paper, June 2008

ERAERA

• 8 discipline clusters

• ERA will not determine block grant allocations of research funding

• Evaluations in each cluster will take 3-4 months

• 3 categories of indicators

• Research activity and intensity

• Research income; research degree load and completions; FTE researcher counts

• Research quality

• Publications analysis (ranked outlets, citation analysis, percentile analysis)

• Research income based on peer review

• Applied Research

• Still to be determined

ERA: AttributionERA: Attribution

• ‘It is proposed that irrespective of the decision on publications, all other indicators will be collected based on institution affiliation because attributing research income and other indicators to staff who have moved would be difficult to monitor and verify’ (AR Consultation Paper)

• Two options being considered:

• staff affiliation at census date (attribution based on where researchers in presently working). May be difficult to collect the data.

• Institution affiliation based on HERDC (HE Research Data Collection): indicators attributed to the institution in which the activity occurred during the reference period. Would take advantage of existing data collection processes within institutions. But the drawback is that it would nor reflect the current of prospective state of affairs within an institution.

ERA ‘Outlet rankings’ERA ‘Outlet rankings’

• More than 17k journals have been ranked across 100 disciplines. More than either Thomson ISI or Scopus (they each rank ~ 15k indexed journals). Done by learned societies and other disciplinary lead bodies.

• Draft ranking list will be released this month.

• Will be used to derive journal sets from which discipline-specific citation benchmarks will be generated. Necessary because the discipline journal sets (‘subject categories’) used by ISI and Scopus are very broad, and contain overlaps. This skews centile analysis when disciplines with different citation practices are included in a single journal set

• Total publication and citation counts for each institution will be assessed, and citation-per-publication rates for each determined. This will be presented to the relevant Research Assessment Committee.

• The ARC will test the use of multiple citation data suppliers to determine the most appropriate supplier for each discipline.

• Outputs which include research students as authors will be analysed separately

ReportingReporting

• Ideally by discipline, to identify areas of national research strength

• By institution: to allow for comparative research strengths across institutions; and

• By academic unit: to provide flexibility for institutions

Dutch METIS systemDutch METIS system

• Research information database

• Offers secured access to universities, research institutes and individual researchers

• Provides management information ‘necessary for the periodical evaluation of the research activities within the institution’

• Permits showcasing by individual researchers

• Includes detailed information on publication outputs, with URLs

• Also includes data on inputs, shown as FTEs

• Used by all13 Dutch universities +KNAW (Academy of Arts & Sciences)

• Linked to all DARE repositories

www.cos.orgwww.cos.org

King’s College Research GatewayKing’s College Research Gateway

Imperial College PWPs systemImperial College PWPs system

Imperial’s Publications systemImperial’s Publications system

The Leiden CourseThe Leiden Course

Leiden benchmarking serviceLeiden benchmarking service