durham researcher development programme 2015-16: bibliometric research indicators

77
Contact Details Academic Liaison Librarian (Researcher Support) [email protected] Phone: +44 (191) 334 2961 Measuring Research Impact: Bibliometric Research Indicators

Upload: jamie-bisset

Post on 18-Jan-2017

84 views

Category:

Education


0 download

TRANSCRIPT

Contact Details

Academic Liaison Librarian (Researcher Support) [email protected] Phone: +44 (191) 334 2961

Measuring Research Impact:

Bibliometric Research Indicators

Learning Outcomes for the session

• Understanding the original purpose and intended uses of

bibliometric research indicators

• Understanding of how some key indicators (JIF, H-index)

are calculated

• Ability to make a judgement as to the appropriateness and

limitations of such indicators

• Ability to use online datasets to view and calculate key

bibliometric measures

• Awareness of some factors which can increase the

visibility and discoverability of your own publications.

Session outline

• Introduction: What can you measure?

• Citations

• Author metrics

• Journal metrics

• Optimising cite-ability

Quick Survey (1)

• How many of you have already published

research papers in journals conference

proceedings?

• How many of you are expected to during the

course of your studies?

• How many of you have already been advised

where to publish, or where not to publish, based

on what are the “best” journals?

What can you measure?

• Article/Book impact

• One measure of the impact of

individual journal articles,

conference proceedings or books,

is the number of times they are

cited by other works.

What can you measure?

• Publication (Journal) impact

• The perceived impact of a specific

academic journal might be

assessed by the number of times

their articles are cited (on average)

and where they are cited.

What can you measure?

• Researcher impact

• The number of outputs and citation

a researcher generates can be an

indicator for the impact of an

individual researcher.

What can you measure?

• Institutional impact

• The prestige of a department or

area of research within an

institution compared to those at

other institutions can be measured

by the sum of individual

researchers ‘impact’.

Impact ?

Are bibliometrics

effective & reliable

measures of

academic impact?

Quick Survey (2)

• Web of Science

• JCR

• Sciverse Scopus

• PoP software

• JIF

• Eigenfactor

• SCImagoJR /

SNIP

• H-index / g-index

• altmetrics

• Sherpa ROMEO

• Sherpa JULIET

• Sherpa FACT

Part 1 Citations

and

Citation

Indices

Citations

• Links between papers that have

something in common

• Building upon, or challenging, research

• Help make a judgement about impact

an article has made

• Sum of citations can be an indication of

the ‘impact’ of an author’s work / a

journal as a collection of articles

Citation indices

• 1955: Eugene Garfield proposed

creating a citation index for science to...

“eliminate the uncritical citation of fraudulent,

incomplete or obsolete data by making it

possible for the conscientious scholar to be

aware of criticisms of earlier papers” Garfield, E (1955) ‘Citation Indexes for Science’

Science, New Series, Vol. 122, No. 3159, pp. 108-111

Citation indices

• 1955: Starts with the idea of measuring the

‘impact’ of journal articles using citations

• 1960s: Science Citation Index developed to

highlight “formal, explicit linkages between

papers that have particular points in common”

• 1975: Journal Citation Reports use Web of

Science data to rank journals within

disciplines

Citation indices

(2008) Taylor and Francis LibSite Newsletter,

issue 9. p. 5

Science subjects

Social-science subjects

Citation indices

“reference lists are held under copyright by

academic publishers which makes tracking

citations impossible”

The death of the reference and the re-use factor (2013) http://figshare.com/blog/The_Death_Of_The_Reference_and_the_reuse_factor/103

Citation indices

• Web of Science

• Provided by Thomson Reuters.

• Includes the Sciences, Social Sciences, Arts &

Humanities & Books Citation Indexes.

• Indexes about 11,800 journals, plus conference

proceedings.

• Approximately 5,000 journals covering arts,

humanities and social sciences.

Citation indices

• Sciverse Scopus

• Launched by Elsevier in 2004.

• Main commercial competition to Web of Science

• Currently indexes c19,000 ‘active’ journals plus

conference proceedings

• Many titles covered by both WoS and Scopus

Citation indices

• Google Scholar

• Pulls data from a much broader range of documents

(eg books, reports, blogs, wider range of publishers)

• Useful for some subjects not covered by Web of

Science

• Citation data: quality and accuracy often

questionned.

Things you can do

• Count citations to an article

• Link to other related articles

• Citation mapping

• Set up citation alerts

• Search for cited references

• See citation reports for authors and journals

Demo Web of Science /

Google Scholar

Over to you • Check Google Scholar and Web of Science for an

article you have already read or are aware of (or

try:

Zong, Y. and Tooley, M. J. (2003) 'A historical record of coastal

floods in Britain : frequencies and associated storm tracks.',

Natural hazards, 29 (1). pp. 13-36.

• Explore and compare the number of citations (if

any) listed

Citation searching

Via Flickr Creative Commons, by © Stuti Sakhalkar. Original available at http://www.flickr.com/photos/theblackcanvas/2945878325/

Citation Metrics

h-index (Hirsch, 2005)

• an author’s number of articles (h) that have

received at least h citations

• a researcher with an h-index of 10 has published

10 articles that have each been cited 10+ times

g-index (Egghe, 2006)

• the highest number (g) of papers that together

received g2 or more citations

• a researcher with a g-index of 10 has published 10

papers that, in total, have been cited at least 100

times

h-index

Author: Smith, J

Has written and published 9 articles (a-i),

which have been cited as follows:

a:3, b:6, c:6, d:2, e:13, f:3, g:0, h:1, i:3

“no. of articles (n) that have received at least n citations”

h-index: what’s in a number?

• Nobel Prize Winner 2013, Peter W

Higgs

• H-index (Google Scholar) = 12

• H-index (Web of Science) = 11

g-index

Author: Smith, J

Has written and published 9 articles (a-i),

which have been cited as follows:

a:3, b:6, c:6, d:2, e:13, f:3, g:0, h:1, i:3

“the highest number (g) of papers that together have received

g2 or more citations”

Demo

Web of Science

Over to you • If you have published 1 or more papers, and if all

are indexed on Web of Science or Google Scholar,

see if you can calculate and compare your h-index

(or try to calculate your g-index)

• ... or try the same for a colleague, supervisor or

head of department.

Author metrics

Via Flickr Creative Commons, by © Stuti Sakhalkar. Original available at http://www.flickr.com/photos/theblackcanvas/2945878325/

Google Scholar – My Citations

• Track citations to your publications

• Check who is citing your publications. Graph your

citations over time. Calculate citation metrics.

• View publications by colleagues or co-authors

• Keep up with their work, view their citation

metrics.

• Appear in Google Scholar Search Results

• Create a public profile that can appear in Google

Scholar when someone searches for your name.

Demo

Google Scholar

– My citations

Author metrics – Issues?

• Author identification

e.g. Professor Gordon Love

A name is not unique

o Prof. Gordon Love, University of California (Earth Science)

o Dr Gordon L Love, Sacramento (Medicine and Health)

o Prof. Gordon Love, Durham University (Physics)

Author metrics – Issues?

• Author identification

e.g. Professor Gordon Love

... So you need a unique identifier (or 3)

o ORCID profile (0000-0001-5137-9434)

o Researcher ID profile (A-3071-2011)

o Google Scholar profile (3xJXtlwAAAAJ)

Demo

ResearcherID

ORCiD

Publish or Perish (PoP) software

• Anne-Wil Harzing (2006), current version

4.25.1 (17th January 2016)

• Aimed at individual researchers

• Analyse own performance using a range of

metrics

• FREE TO DOWNLOAD (Windows, Apple OS

X, GNU/Linux) and FREE TRAINING

MATERIAL

• http://www.harzing.com/pop_win.htm

Part 3 Journal

Metrics

Journal Citation Reports (JCRs)

• JCRs – annual publication of journals and

their impact factors.

• Over 10,800 titles, across 232 disciplines

have JIFs in 2015 editions

• A journal that is cited once, on average, for

each article published has an JIF of 1.

Journal Citation Reports (JCRs)

• 2015 edition (published June 2015):

• is JCR year 2014 (providing 2014 Journal

Impact Factor (JIF) metrics)

• counting citation data from 2014

• for articles published in 2012/13

Journal Impact Factor (JIF)

Citations in 2014 to all

articles published by Journal

X in 2012 & 2013

Number of articles that

were published in Journal X

in 2012 & 2013

Journal

X’s

2014

impact

factor

=

Journal Impact Factor (JIF)

Citations in 2014 (in journals

indexed in Web of Science) to

all articles published by

Journal X in 2012 & 2013

Number of articles (of a type

deemed to be citeable) that

were published in Journal X

in 2012 & 2013

Journal

X’s

2014

impact

factor

=

Journal Ranking

Journal Ranking

Demo

JCRs

Journal Impact Factors

Other journal impact metrics

• Eigenfactor

• Calculates weightings based on ‘where’

the citation came from

• Also takes into account the ‘size’ of a

journal, to measure the ‘total importance of

a journal’

• http://www.eigenfactor.org/ (1997-2013)

Other journal impact metrics

• SCImago

• Based on data from Scopus (rather than

Web of Science)

• http://www.scimagojr.com/journalrank.php

Over to you • Explore the JCRs to identify relevant subject

categories for your field of research.

• Identify journals and compare JIFs and Eigenfactor

Scores

Journal metrics

Via Flickr Creative Commons, by © Stuti Sakhalkar. Original available at http://www.flickr.com/photos/theblackcanvas/2945878325/

Issues with journal metrics

• Author identification

• Citation cultures vary across disciplines

• Publication cultures vary across disciplines

• Research careers have different stages

• Citations ≠ confirmation of excellence

• Scholarly communication is evolving…

• Blogs, twitter, data and open access

repositories

Optimising the cite-ability of an article

• Title & Abstract

• Discoverability

• Length

• Reference List

• Journal / means of publication

• Journal Impact Factors

• Open Access Publication

Title and abstract - Discoverability

“Construct a clear, descriptive title”

• describe what the research is about

• consider what readers might be looking for

“Re-iterate key phrases in the abstract”

• improves potential search ranking

• ‘aids’ human decision-making

Publisher guidelines (Wiley-Blackwell)

Title and abstract - Discoverability

Easier to find

=

More likely to be read

=

May translate to increased citations (but still

depends on the quality of the research)

“Ever since the British colonists in Australia became aware of the

disappearance of the indigenous peoples in the 1830s, they have

contrived to excuse themselves by pointing to the effects of disease and

displacement. Many colonists called for the extermination of Aborigines

when they impeded settlement by offering resistance, yet there was no

widespread public acknowledgement of this as a policy until the later

1960s, when a critical school of historians began serious investigations

of frontier violence. Their efforts received official endorsement in the

1990s, but profound cultural barriers prevent the development of a

general awareness of this. Conservative and right-wing figures continue

to play down the gravity of what transpired. These two aspects

of Australian public memory are central to the political humanisation of

the country.”

Australia’s Forgotten

Victims

Genocide and Holocaust

Consciousness in Australia “Ever since the British colonists in Australia became aware of the

disappearance of the indigenous peoples in the 1830s, they have contrived

to excuse themselves by pointing to the effects of disease and

displacement. Yet although genocide was not a term used in the nineteenth

century, extermination was, and many colonists called for the extermination

of Aborigines when they impeded settlement by offering

resistance. Consciousness of genocide was suppressed during the

twentieth century until the later 1960s, when a critical school of historians

began serious investigations of frontier violence. Their efforts received

official endorsement in the 1990s, but profound cultural barriers prevent the

development of a general genocide consciousness. One of these

is Holocaust consciousness, which is used by conservative and right-wing

figures to play down the gravity of what transpired in Australia. These two

aspects of Australian public memory are central to the political

humanisation of the country. ”

Title and abstract - Length

• Didegah, F. and Thelwall, M. (2013) - Looked at 16,058 Biology/Biochemistry articles, 16,378

Chemistry articles and 15,392 Social Sciences articles all

covered by Web of Science.

• “abstract length significantly associates with

increased citation impact in all fields”

• “the number of keywords and the title length

statistically associate with decreased citations”

Didegah, F. and Thelwall, M. (2013) “Which factors help authors produce the highest impact research?

Collaboration, journal and document properties” Journal of Informetrics 7: 861-873. Available at

http://dx.doi.org/10.1016/j.joi.2013.08.006

Title and abstract - Length

Reference List

• “The impact [factor] and the number of cited

references are … significant determinants of

increased citation impact”

Didegah, F. and Thelwall, M. (2013) “Which factors help authors produce the highest impact research?

Collaboration, journal and document properties” Journal of Informetrics 7: 861-873. Available at

http://dx.doi.org/10.1016/j.joi.2013.08.006

• Consistent with studies in fields including

psychology, medicine, chemistry, physics,

engineering…

Place of Publication: JIF

• “… the JIF is the main determinant of article citation

impact”

Didegah, F. and Thelwall, M. (2013) “Which factors help authors produce the highest impact research?

Collaboration, journal and document properties” Journal of Informetrics 7: 861-873. Available at

http://dx.doi.org/10.1016/j.joi.2013.08.006

• “… the he impact factor of the journal in which

articles appeared was the primary predictor of the

citations that they accrued”

Haslam, N. and Koval, P. (2010) “Predicting long-term citation impact in articles in social and personality

psychology” Psychological Reports 106(3) Jun 2010. Available at http://dx.doi.org/10.2466/pr0.106.3.891-900

Place of Publication: JIF

• “…a small but significant correlation between journal

rank and future citations can be observed … stems

from visibility effects due to the influence of the IF on

reading habits (Lozano et al., 2012), rather than from

factors intrinsic to the published articles … [but] the

correlation is so weak that it cannot alone account for

the strong correlation between retractions and

journal rank”

Björn Brembs, Katherine Button and Marcus Munafò (2013) “Deep impact: unintended consequences of journal

rank” Frontiers in Human Neuroscience 7: 291. Available at http://dx.doi.org/10.3389/fnhum.2013.00291

Citing

Lozano, G. A., Larivière, V., and Gingras, Y. (2012). “The weakening relationship between the impact factor and

papers' citations in the digital age” J. Am. Soc. Inf. Sci. Technol. 63, 2140–2145. Available at

http://dx.doi.org/10.1002/asi.22731

Place of Publication: JIF

Fang, F. C., & Casadevall, A. (2011). Retracted Science and the Retraction Index

. Infection and Immunity, 79(10), 3855–3859. http://doi.org/10.1128/IAI.05661-11

Impact ?

Place of Publication: Open Access

• Boost the potential visibility and access of research,

to a wider audience

• Removes research from behind paywall barriers for

existing audience

• Enables authors to retain their rights and more easily

share via social media, email etc.

Place of Publication: Open Access

• 4633 articles across ecology, applied

mathematics, sociology and economics.

• 2280 were open access, and had an

average citation count of 9.04

• 2353 were subscriptions journals, and had

an average citation count of 5.76.

Norris, M. (2008) “The citation advantage of open access articles” Thesis. Available at

https://dspace.lboro.ac.uk/2134/4089

Place of Publication: Open Access

SPARC Europe: Open Access Citation Advantage Service

http://sparceurope.org/oaca/

Total number of studies so far 70

Studies that found a citation advantage 46

Studies that found no citation advantage 17

Studies that were inconclusive, found non-

significant data or measured other

things than citation advantage for articles

7

Place of Publication: Open Access

Terras, M. (2011) “What happens when you tweet an Open Access Paper” Melissa

Terras’ Blog. Available at http://melissaterras.blogspot.co.uk/2011/11/what-happens-

when-you-tweet-open-access.html

By October 2012, the OA

version had seen nearly 3

times more downloads than

the version sitting behind a

subscription paywall.

Demo

Sherpa Romeo

Sherpa FACT

Over to you • (very briefly) For any of the journals you have

previously looked at (eg to se their JIF), use

Sherpa Romeo (or Sherpa FACT if your research

is funded) to identify your open access options (if

you can interpret Sherpa!)

Checking OA options

Via Flickr Creative Commons, by © Stuti Sakhalkar. Original available at http://www.flickr.com/photos/theblackcanvas/2945878325/

Other suggestions…

• Dr Michael Taylor, Dept. Earth Sciences (Bristol)

- Discusses strengths and weaknesses of his already

published article titles.

- http://tinyurl.com/k6dhcac

- http://tinyurl.com/k7o9msc

- avoid vague words / weak puns

- NEGATIVE “it’s 12 characters too long to tweet”

- POSITIVE “ the title strongly implies the conclusion”

- POSITIVE “ Short, appealing and (hopefully) funny.”

Beyond Bibliometrics…

• Altmetrics…

Learning Outcomes for the session

• Understanding the original purpose and intended uses of

bibliometric research indicators

• Understanding of how some key indicators (JIF, H-index)

are calculated

• Ability to make a judgement as to the appropriateness and

limitations of such indicators

• Ability to use online datasets to view and calculate key

bibliometric measures

• Awareness of some factors which can increase the

visibility and discoverability of your own publications.

Image Credits

[Slide 5] Via Flickr Creative Commons, by demonsub. Original available at http://www.flickr.com/photos/66397367@N05/6993902269

[Slide 13] Via Flickr Creative Commons, by Kirsty Andrews. Original available at http://www.flickr.com/photos/47745961@N08/5169765739

[Slide 23, 35, 50, 71] Via Flickr Creative Commons, by © Stuti Sakhalkar. Original at http://www.flickr.com/photos/theblackcanvas/2945878325/

[Slide 40] Via Flickr Creative Commons, by Alan Cleaver. Original available at http://www.flickr.com/photos/11121568@N06/4122176776/

[Slide 22] ‘Vitae®, © 2010 Careers Research and Advisory Centre (CRAC) Limited‘ Available at www.vitae.ac.uk/rdf

Measuring

Researcher

Development Vitae Researcher Development Framework [see image credits]

Durham University Library: Supporting your research

Contact Details

Academic Liaison Librarian (Researcher Support) [email protected] Phone: +44 (191) 334 2961