BibliometricsBibliometrics: An : An insiderinsider’’s Views View

Sabina Robertson, University of MelbourneSabina Robertson, University of MelbourneLynne Horwood, University of MelbourneLynne Horwood, University of Melbourne

Jenny Cameron, Victoria UniversityJenny Cameron, Victoria University

CRIG: Information Literacy in Interesting Times CRIG: Information Literacy in Interesting Times November 25, 2009November 25, 2009

Presentation OverviewPresentation Overview

Part One: Context (Sabina Robertson)Part One: Context (Sabina Robertson)

Part Two: Measures, products and Part Two: Measures, products and applications (Lynne Horwood)applications (Lynne Horwood)

Part Three: Models of research support: Part Three: Models of research support: (Jenny Cameron)(Jenny Cameron)

Part One: ContextPart One: Context Government policies setting Government policies setting

the research scenethe research scene

Rankings of universitiesRankings of universities•• Promotion, awarding of Promotion, awarding of

research fundsresearch funds

Our academics responding Our academics responding to new research environmentto new research environment

ERAERA•• issues issues –– different disciplines different disciplines

need different measuresneed different measures

Government imperativesGovernment imperatives Positioning Australia in Positioning Australia in

the International arenathe International arena Competition from India & Competition from India &

China China Identifying areas for Identifying areas for

research investmentresearch investment Attracting Researchers to Attracting Researchers to

AustraliaAustralia Universities accountable Universities accountable

for R & D fundingfor R & D funding

Perspective from a University Perspective from a University Vice Chancellor Vice Chancellor

Rigorous monitoring and reviews:Rigorous monitoring and reviews:

InternationalInternational ranking systems: Shanghai Jiao Tong World Universities & ranking systems: Shanghai Jiao Tong World Universities &

Times Higher Education RankingsTimes Higher Education Rankings

National National •• Performance reports, National protocols & auditsPerformance reports, National protocols & audits

LocalLocal•• Internal faculty, departmental and divisional reviews with exterInternal faculty, departmental and divisional reviews with external nal

reviewersreviewers

ReputationReputation•• Individual reputations of institutions built up over long periodIndividual reputations of institutions built up over long periods of times of time•• Includes community engagementIncludes community engagement

High Quality University SystemsHigh Quality University SystemsCurrent review and monitoring systems have achieved:Current review and monitoring systems have achieved:

Universities aspiring to quality and excellence in:Universities aspiring to quality and excellence in: ResearchResearch InnovationInnovation Student learningStudent learning TeachingTeaching Community and industry engagementCommunity and industry engagement

Universities actively pursuing their different missionsUniversities actively pursuing their different missions

Demand for university placesDemand for university places

Demand from government and industry for university researchDemand from government and industry for university research

Australian Universities: Australian Universities: International RankingsInternational Rankings

Australia has an international reputation of having a strong uniAustralia has an international reputation of having a strong universityversitySystem in research; teaching and learning; international EducatiSystem in research; teaching and learning; international Educationon

Reflected in Universities in International Rankings system:Reflected in Universities in International Rankings system:

Shanghai Jiao TongShanghai Jiao Tong 17 of 38 Australian Universities in Top 500;17 of 38 Australian Universities in Top 500; 7 in Top 2007 in Top 200

Times Higher Education (THE)Times Higher Education (THE) 12 Australian Universities in Top 20012 Australian Universities in Top 200

Australia performs highly on international rankings.Australia performs highly on international rankings.

World University Rankings: World University Rankings: indicators used indicators used

The rankings derived from calculations based on The rankings derived from calculations based on data gathered from the following indicators:data gathered from the following indicators: Academic peer review Academic peer review Employer review Employer review International academic staff ratio International academic staff ratio International student ratio International student ratio Student : academic staff ratio Student : academic staff ratio

Citations per academic staff member Citations per academic staff member (citation data supplied by Scopus) (citation data supplied by Scopus)

Role of The Australian Research Role of The Australian Research Council (ARC)Council (ARC)

ARC roles:ARC roles: A statutory authority within the Australian GovernmentA statutory authority within the Australian Government’’s s

Innovation, Industry, Science and Research portfolio.Innovation, Industry, Science and Research portfolio.

Advises the Government on research matters and Advises the Government on research matters and manages the National Competitive Grants Program manages the National Competitive Grants Program (NCGP) . (NCGP) .

Responsible for the Australian research quality and Responsible for the Australian research quality and evaluation system evaluation system –– the the Excellence in Research for Excellence in Research for AustraliaAustralia (ERA) initiative.(ERA) initiative.

Excellence in Research for Excellence in Research for Australia (ERA): the experienceAustralia (ERA): the experience

Cluster Trials 2009: Cluster Trials 2009: Cluster 1: Cluster 1:

•• Physical, Chemical & Physical, Chemical & Earth Sciences PCE)Earth Sciences PCE)

Cluster 2:Cluster 2:•• Humanities and Creative Humanities and Creative

Arts (HCA)Arts (HCA)

Data collection rangesData collection ranges

Submission data for ERA was collected for the Submission data for ERA was collected for the following reference periods:following reference periods:

Data Type Reference Period YearsData Type Reference Period YearsResearch Outputs 1 January 2002Research Outputs 1 January 2002––31 December 31 December

2007 (6 years)2007 (6 years)Research Income 1 January 2005Research Income 1 January 2005––31 December 31 December

2007 (3 years)2007 (3 years)Applied Measures 1 January 2005Applied Measures 1 January 2005––31 December 31 December

2007 (3 years)2007 (3 years)

ERA: Eligible Research ERA: Eligible Research OutputOutput

For the PCE and HCA clusters, the common For the PCE and HCA clusters, the common eligible research output types were:eligible research output types were:

Books: Authored ResearchBooks: Authored Research Book: Chapters in Research BookBook: Chapters in Research Book Journal Articles: Refereed, Scholarly Journal Articles: Refereed, Scholarly

JournalJournal Conference Publication: Full Paper Conference Publication: Full Paper

Refereed.Refereed.

Humanities and Creative Arts (HCA) Humanities and Creative Arts (HCA) cluster criteria. cluster criteria.

Criteria used to select 20% of research output as Criteria used to select 20% of research output as follows:follows:

articles that are published in articles that are published in A*A* or or A A category category journals journals ERA journal listsERA journal lists

books that discipline heads consider to have had a books that discipline heads consider to have had a major impact in the field major impact in the field -- not limited by publishernot limited by publisher

book chapters book chapters –– assessed on criteria set by ERA assessed on criteria set by ERA Committee of expertsCommittee of experts

Use of Information gathered from Use of Information gathered from ERA cluster trialsERA cluster trials

The Australian Research Council (ARC) will The Australian Research Council (ARC) will publicly release outcomes of the evaluations publicly release outcomes of the evaluations for Clusters One and Two aggregated to the for Clusters One and Two aggregated to the national level but not broken down to the national level but not broken down to the level of individual institutions.level of individual institutions.

ARC will also make available to individual ARC will also make available to individual institutions analyses of their own disciplines. institutions analyses of their own disciplines. These individual analyses will not be publicly These individual analyses will not be publicly released by ARC. released by ARC.

University of Melbourne Research & University of Melbourne Research & Innovation OfficeInnovation Office

University of Melbourne Research University of Melbourne Research PerformancePerformance

An AcademicAn Academic’’s response: Dr s response: Dr Christopher MarshallChristopher Marshall

ERA ranking: A journal

“A” ranked ERA journal“A” ranked ERA journal

“A” ranked ERA journal

Measures, products and usesMeasures, products and uses

Evaluating researchers Evaluating researchers

Measuring citation impact (research Measuring citation impact (research impact)impact)

Evaluating journalsEvaluating journals

hh--indexindex

The The hh--index was developed in 2005 by index was developed in 2005 by Jorge Hirsch, a condensedJorge Hirsch, a condensed--matter matter physicist at the University of California in physicist at the University of California in San Diego.San Diego. HirschHirsch’’s aim was to qualify the s aim was to qualify the impact and quantity of an individual impact and quantity of an individual scientistscientist’’s research output.s research output.

HirschHirsch’’s original paper appears at: s original paper appears at: http://arxiv.org/abs/physics/0508025http://arxiv.org/abs/physics/0508025

hh--index and citation patternsindex and citation patterns

Calculating the Calculating the hh--indexindex

hh--index and index and Web of ScienceWeb of Science

Google Scholar Universal Google Scholar Universal GadgetGadget

A Google Scholar Universal Gadget A Google Scholar Universal Gadget enables users to search for the total enables users to search for the total number of citations of number of citations of author(sauthor(s). It ). It provides a total citation count, total provides a total citation count, total number of cited publications and Jorge E. number of cited publications and Jorge E. HirschHirsch’’s s hh--Index.Index.

Sources for Sources for hh--index calculation index calculation ––comparison of the three productscomparison of the three products

Web of ScienceWeb of Science117 papers 117 papers hh--index = 31index = 31

Google ScholarGoogle Scholar158 publications158 publicationshh--index =34index =34

ScopusScopus84 papers84 papershh--index = 25index = 25

Evaluating researchersEvaluating researchers

Citation analysisCitation analysis

Examines the elements of bibliographic Examines the elements of bibliographic recordsrecords

Studies relationships between citationsStudies relationships between citationsMeasures the amount of activity in a Measures the amount of activity in a

subject category, journal, country or topic subject category, journal, country or topic

Citation counts and 3 productsCitation counts and 3 products

McGorryMcGorry, P. D., Edwards, J., , P. D., Edwards, J., MihalopoulosMihalopoulos, C., , C., HarriganHarrigan, S. M., & Jackson, H. J. (1996). EPPIC: , S. M., & Jackson, H. J. (1996). EPPIC: An evolving system of early detection and An evolving system of early detection and optimal management. optimal management. Schizophrenia Bulletin, Schizophrenia Bulletin, 2222(2), 305(2), 305--326.326.

Cited Cited 460460 times (times (Google ScholarGoogle Scholar))Cited Cited 339339 times (times (Web of ScienceWeb of Science))Cited Cited 349349 times (times (ScopusScopus))

ScopusScopus : Arts and Humanities : Arts and Humanities coverage increased in 2009coverage increased in 2009

Subjects with the most journals include Subjects with the most journals include literature and literary theory (30% of new titles) literature and literary theory (30% of new titles)

general arts and humanities (22%) general arts and humanities (22%) history (17%) history (17%) visual/performing arts (16%) visual/performing arts (16%)

More than half (60%) are published from the EMEA More than half (60%) are published from the EMEA (Europe, Middle East and Africa) region, 38% from the (Europe, Middle East and Africa) region, 38% from the Americas and 2% are from AsiaAmericas and 2% are from Asia--Pacific.Pacific.

ScopusScopus –– title, abstract, keyword title, abstract, keyword searchsearch

ScopusScopus Citation trackerCitation tracker

Alternatives to Alternatives to Web of ScienceWeb of Science and and ScopusScopus

CiteSeerXCiteSeerX (Information Technology) (Information Technology) Google Scholar (Multidisciplinary) Google Scholar (Multidisciplinary) ScienceDirect(MultidisciplinaryScienceDirect(Multidisciplinary)) Business Source Premier (Business Business Source Premier (Business

research database) research database)

Most cited computer science Most cited computer science articlesarticles

ScienceDirectScienceDirect

Business Source PremierBusiness Source Premier

Researcher IDResearcher ID

http://http://researchanalytics.thomsonreuters.comresearchanalytics.thomsonreuters.com//

Tracking hot papersTracking hot papers

Science WatchScience Watch Essential Science Essential Science

IndicatorsIndicators Highly Highly Cited.comCited.com

ISIHighlycited.comISIHighlycited.com

Essential Science IndicatorsEssential Science Indicators

Journal evaluationJournal evaluation

To which journal should you submit your To which journal should you submit your high profile research for publication?high profile research for publication?

RejectaRejecta MathematicaMathematica

Journal for papers Journal for papers rejected elsewhererejected elsewhere

Measuring journal quality Measuring journal quality –– journal journal ranking systemsranking systems

Impact factor is one measureImpact factor is one measure Journal rankings by subject groupingJournal rankings by subject grouping ARC list of journals ARC list of journals CORE CORE –– computing journalscomputing journals ERIH ERIH ––humanitieshumanities BARDsNETBARDsNET -- economicseconomics

ScopusScopus –– Journal Journal AnalyzerAnalyzer

Journal Citation ReportsJournal Citation Reports

Journal Citation ReportsJournal Citation Reports

5 year impact factor as a new measure5 year impact factor as a new measure

Impact factorsImpact factors

Impact factor criticismImpact factor criticism““Beware the tyranny of impact factorsBeware the tyranny of impact factors””..““Challenging the tyranny of impact factorsChallenging the tyranny of impact factors””..““Corruption of journal impact factorsCorruption of journal impact factors””..““Impact factors Impact factors ‘‘flawed, misleading and unfairflawed, misleading and unfair’”’”..““Journal impact factors for the individual scientist: an unnecessJournal impact factors for the individual scientist: an unnecessary evilary evil””..““LetLet’’s dump the factorss dump the factors””..““Nightmare impact factorNightmare impact factor””..““The dreaded impact factor is back to haunt us!The dreaded impact factor is back to haunt us!””..““The impact factor The impact factor –– ‘‘Misleading, unscientific and unjustMisleading, unscientific and unjust’”’”..““The impact factor The impact factor –– what it is and where it is uselesswhat it is and where it is useless””..““The journal The journal ‘‘impact factorimpact factor’’: a misnamed, misleading, misused measure: a misnamed, misleading, misused measure””..““The malign influence of impact factorsThe malign influence of impact factors””..““The perfidy of impact factorsThe perfidy of impact factors””..““The tyranny of the impact factorThe tyranny of the impact factor””..““Worshiping false idols: the impact factor dilemmaWorshiping false idols: the impact factor dilemma””..

Food for thoughtFood for thought

Elite journals wield huge influence and Elite journals wield huge influence and control over scientistscontrol over scientists’’ lives (lives (CorbynCorbyn 2009)2009)

Sir John Sulston US Nobel Laureate :Sir John Sulston US Nobel Laureate : Pressure to publish prematurelyPressure to publish prematurely His work not published in a top journalHis work not published in a top journal Treadmill of re submissionsTreadmill of re submissions Provides citations for many, but does little for Provides citations for many, but does little for

scientific enterprise, eg long author lists, but scientific enterprise, eg long author lists, but some may have marginal contributionsome may have marginal contribution

Part Three: Part Three: Models of research Models of research supportsupport

Understanding the contextUnderstanding the context

Understanding the usesUnderstanding the uses

Levels of supportLevels of support

Libraries responding to the Libraries responding to the research environmentresearch environment

Librarians need to understand the context:Librarians need to understand the context: Research funding highly competitiveResearch funding highly competitive Culture of evaluationCulture of evaluation Imperative to be Imperative to be ‘‘research activeresearch active’’ BibliometricsBibliometrics can be confusing!can be confusing!

Seizing the momentSeizing the moment

BibliometricsBibliometrics –– a perfect opportunity to a perfect opportunity to reconnect with academic staffreconnect with academic staff

They are looking to us for guidanceThey are looking to us for guidance Show our expertise (and can spill over to other Show our expertise (and can spill over to other

areas, eg trust to embed IL)areas, eg trust to embed IL) But we need to understand the uses (in detail!) But we need to understand the uses (in detail!)

that they are making of that they are making of bibliometricsbibliometrics Professional development for library staffProfessional development for library staff

Use of metricsUse of metrics

For grant applicationsFor grant applications Academic promotionAcademic promotion RecruitmentRecruitment Research assessment (school, research centre, Research assessment (school, research centre,

whole university)whole university) Where to publishWhere to publish

One use: ARC grant applicationOne use: ARC grant application

Researchers need to nominate their ten careerResearchers need to nominate their ten career--best publications: best publications:

Can be up to 3 pages, Can be up to 3 pages, ieie paragraph per itemparagraph per item Say why each one is best (they must FIGJAM to Say why each one is best (they must FIGJAM to

be competitive!)be competitive!)

Levels of supportLevels of support

Intensive: UNSWIntensive: UNSW SelfSelf--help: Vic Unihelp: Vic Uni Something in between?Something in between?

Intensive supportIntensive support

RIMS at UNSWRIMS at UNSW Restructure to create Research Impact Restructure to create Research Impact

Measurement ServiceMeasurement Service Produce Produce ‘‘Research Impact StatementResearch Impact Statement’’ and more and more

specialised specialised ‘‘Grant Application StatementGrant Application Statement’’ for for academicsacademics

7 EFT staff / 50% of their time spent on creating 7 EFT staff / 50% of their time spent on creating citation reports for researcherscitation reports for researchers

SelfSelf--helphelp

Train the researcher to do it themselves (VU)Train the researcher to do it themselves (VU) Develop, advertise and deliver Develop, advertise and deliver ‘‘Research Research

QualityQuality’’ or or ‘‘Citation TrackingCitation Tracking’’ workshopsworkshops Timed to critical dates Timed to critical dates ieie grant applications or grant applications or

academic promotions applications due soonacademic promotions applications due soon Collaborate with University Research Office, coCollaborate with University Research Office, co--

presenterspresenters Sessions at different campusesSessions at different campuses Wording used to market sessions very importantWording used to market sessions very important

Something in between?Something in between?

Ad hoc intensive support to individuals (even Ad hoc intensive support to individuals (even with a self help model this happens at VU) with a self help model this happens at VU)

Intensive support to target groupsIntensive support to target groups ……combined with workshopscombined with workshops Good web pagesGood web pages…… some examplessome examples

UQUQ

UNSWUNSW

UWSUWS

Issues re supportIssues re support

Intensive Intensive -- full full ‘‘research impactresearch impact’’ report:report: Sustainability?Sustainability? Responsibility with the librarian to corroborate Responsibility with the librarian to corroborate

from multiple sources and provide accurate data from multiple sources and provide accurate data (this is what is SO time consuming)(this is what is SO time consuming)

It is not always clear what measures should be It is not always clear what measures should be used in which context used in which context –– shouldnshouldn’’t the researcher t the researcher decide?decide?

But UNSW service has elevated the libraryBut UNSW service has elevated the library’’s s profileprofile

Issues re supportIssues re support

Ad hoc:Ad hoc: Inequitable? CanInequitable? Can’’t do for everyonet do for everyone Does everyone know about this service?Does everyone know about this service? People (managers/researchers) donPeople (managers/researchers) don’’t know just t know just

how time intensive it is (unrecognized)how time intensive it is (unrecognized)

Self help:Self help: Academics are time poorAcademics are time poor And may leave a workshop still confusedAnd may leave a workshop still confused

W h a t a r e y o u d o in g ?W h a t a r e y o u d o in g ?

How much effort should you go to?How much effort should you go to?

Workshop discussionWorkshop discussion

University of Melbourne:University of Melbourne: BibliometricsBibliometrics workshops for library staff; RHD workshops for library staff; RHD

students; training for supervisorsstudents; training for supervisors Staff assist with grant applications; ERAStaff assist with grant applications; ERA DonDon’’t call it a class, call it a forum/seminart call it a class, call it a forum/seminarDeakin University:Deakin University: Developing a research support strategy Developing a research support strategy

framework: framework: bibliometricsbibliometrics just a part of thisjust a part of this

Workshop discussionWorkshop discussionRMIT:RMIT: Individual support from liaison librariansIndividual support from liaison librarians Citation tracking covered briefly in Citation tracking covered briefly in postgradpostgrad

sessions (for Eng/sessions (for Eng/SciSci)) Developing a research support planDeveloping a research support planOther issues:Other issues: Combining staff & students in single session Combining staff & students in single session

trickytricky Some admitted at end of workshop that they Some admitted at end of workshop that they

hadnhadn’’t focussed in this area yett focussed in this area yet

Initiatives Initiatives

Positioning librarians to support researchPositioning librarians to support research Role of CRIG in supporting developments Role of CRIG in supporting developments

in research supportin research support RecommendationRecommendation

Establishment of regular Establishment of regular forafora librarians librarians working in the field of research supportworking in the field of research support