Evaluation of journals based on bibliometric measures

Stefan Carlstein January 14, 2010

ISI Web of Science (WoS)

• ISI WoS is provided by Thomson Reuters• The three main databases of WoS:

Science Citation Index (coverage 1945-)Social Science Citation Index (coverage 1956-)Arts & Humanities Citation Index (coverage 1975-)

• Approximately 9,300 indexed high impact journals (including Open Access journals)

• Used by The Swedish Research Council (Vetenskapsrådet) for the allocation of grants to Swedish universities and university colleges

Subject coverage in ISI Web of Science (%)

Chemist

ry

Biomed

icine

Physic

s

Medici

ne &

pharm

acolo

gy

Biolog

y

Mathem

atics

Techn

ology

Odonto

logy

Econo

mics

Psych

ology

Other s

cocia

l scie

nces

Human

ities

0

10

20

30

40

50

60

70

80

90

100

Source: Vetenskapsrådet (2009). Biblimetrisk indikator som underlag för medelsfördelning

Journal Citation Reports (JCR)• Covers citation data from more than 8,000 scholarly and technical journals.• Covers approximately 227 disciplines.• Two editions:

The science edition covers over 6,500 journalsThe social sciences edition covers over 1,900 journals

• JCR allows you to see:The most frequently cited journals in a fieldThe highest impact journals in a fieldLargest journals in a field

• Cited and citing journal statistics from 1997 forward• Includes measures like: the Journal Impact Factor (JIF), Immediacy Index,

Cited Half-Life, and also Eigenfactor and Article Influence (2007 and later)

The impact factor (IF)

• The impact factor (IF) is used as a measure of the relative importance of a journal within its field – also called Journal Impact Factor (JIF).

• Journals with higher impact factors are assessed as more important than those who have received a lower impact.

• The calculation of the impact factor is based on citations.

• A citation indicates that an article has influenced an author, and if an article is cited often it would indicate that it has had a strong influence on the field and the research community.

• Created and originally used by Eugene Garfield to help select journals that were to be covered and indexed in Science Citation Index (the Web of Science database).

Why and when do people cite?1. Paying homage to pioneers2. Giving credit for related work (homage to peers)3. Identifying methodology, eqipment, etc.4. Providing background reading5. Correcting one’s own work6. Correcting the work of others7. Criticizing previous work8. Substantiating claims9. Alerting researchers to forthcoming work10. Providing leads to poorly disseminated, poorly indexed, or uncited work11. Authenticating data and classes of fact - physical constants, etc.12. Identifying original publications in which an idea or concept was discussed13. Identifying the original publication describing an eponymic concept or term14. Disclaiming work or ideas of others (negative claims)15. Disputing priority claims of others (negative homage)

Source: Garfield, E. (1996). When to cite. Library Quarterly, 66(4), 449-458

Measures of the scientific quality of journals (1)

Journal Impact factor (JIF)The average number of times articles from the journal published in the past two years have been cited in the JCR year (the edition of JCR = the year of measuring).Calculated by dividing the number of citations in the JCR year by the total number of articles published in the two previous years.Also: 5-Year Journal Impact Factor, calculated the same way as above but using articles published in the past five years instead of two.

Example:A = the number of times articles published in 2006 and 2007 were cited by

indexed journals during 2008 (=JCR year)B = the total number of “citable items" published in 2006 and 2007.

2008 impact factor = A / B

Example: 2008 impact factor for JPIM

JIF = A/B (citations/number of articles)

An Impact Factor of 1.0 means that, on average, the articles published one or two years ago have been cited ONE time.

An Impact Factor of 2.5 means that, on average, the articles published one or two years ago have been cited TWO AND A HALF times.

Citing articles may be from the same journal - most citing articles are from different journals.

2006 2007 2008 (JCR year)

Criticisms of the Impact Factor

Validity• Discipline-dependent• Not a normal distribution• Author self-citations (although not necessarily a sign of low quality)

Manipulation• Journals can adopt editorial policies that increase the impact factor

Misuse• Misused to evaluate individual articles or researchers

The skewed distribution of citations – example journal

Source: Cross (2009). Impact factors – the basics. In The e-resources management handbook.

The skewed distribution of citations - all articles

Graph based on Karolinska Institutet world data acquired from Thomson Reuters, Inc.

Measures of the scientific quality of journals (2)

The journal Immediacy IndexIndicates how quickly articles in a journal are cited.

• The immediacy index is calculated by dividing the number of citations to articles published in a given year by the number of articles published in that year.

• The higher the index the quicker the articles are cited.• The Immediacy Index tends to discount the advantage of large journals over small

ones. • However, frequently issued journals may have an advantage because an article

published early in the year has a better chance of being cited than one published later in the year.

• Many publications that publish infrequently or late in the year have low Immediacy Indexes.

Measures of the scientific quality of journals (3)Journal Cited Half-Life

The median age of the articles that were cited in Journal Citation Reports each year. For example, if a journal's half-life in 2005 is 5, that means the citations from 2001-2005 are half of all the citations from that journal in 2005, and the other half of the citations precede 2001.

Aggregate measures are calculated in the same way as their journal counterparts although aim at whole categories instead of specific journals:

• Aggregate Impact Factor• Aggregate Immediacy Index• Aggregate Cited Half-Life

And also:• Median Impact Factor (the median value of all Journal Impact Factors in the

subject category).In JCR you choose View Category Data instead of View Journal Data.

Measures of the scientific quality of journals (4)

The H index“A scientist has index h if h of [his/her] Np papers have at least h citations each,

and the other (Np − h) papers have at most h citations each.”Source: Hirsch (2005). An index to quantify an individual scientific research output. PNAS, 102(46), 16569–16572.

Ranking order - articles Number of citations

1 17

2 10

3 8

4 8

5 7

6 6

7 6

8 4

9 4

10 4

6 publications that have at least 6 citations each

The rest of the publications have 6 citations each at the most

Measures of the scientific quality of journals (5)Eigenfactor and Article Influence• Use citation data to assess and track the influence of a journal in relation to other journals.• Metrics in JCR only available for JCR years 2007 and later.• Aim to identify the most influential journals – where a journal is considered to be influential if it is

cited more often by other influential journals.• Larger journals will have have more citations and therefore larger Eigenfactor Scores.• Measure the importance of a citation by the influence of the citing journal divided by the total

number of citations appearing in that journal• Similar to Google’s PageRank.• A network structure of citations.

www.eigenfactor.org/whyeigenfactor.htm

Measures of the scientific quality of journals (5)

Eigenfactor Score

Measures the number of times articles from the journal published in the past five years have been cited in the JCR year.Similar to the Impact Factor - essentially a ratio of number of citations to total number of articles. But, unlike the Impact Factor, the Eigenfactor Score:

• Counts citations to journals in both the sciences and social sciences.• Eliminates self-citations. Every reference from one article in a journal to another

article from the same journal is discounted.• Weights each reference according to a stochastic measure of the amount of time

researchers spend reading the journal.• Rates journals according to the number of incoming citations, with citations from

highly-ranked journals weighted to make a larger contribution to the Eigenfactor than those from poorly-ranked journals.

Measures of the scientific quality of journals (5)

Eigenfactor Score continued

“The algorithm effectively calculates the trajectory of a hypothetical ‘random researcher’ who behaves as follows. Our random researcher begins by going to the library and selecting a journal article at random. After reading the article, she selects at random one of the citations from the article. She then proceeds to the cited work and reads a random article there. She selects a new citation from this article, and follows that citation to her next journal volume. The researcher does this ad infinitum. Since we lack the time to carry out this experiment in practice, Eigenfactor uses mathematics to stimulate this process.”

Source: Bergstrom (2007). Eigenfactor: measuring the value and prestige of scholarly journals. C&RL News, 68(5).

Measures of the scientific quality of journals (6)

Article Influence

Measures the relative importance of the journal on a per-article basis.It is the journal's Eigenfactor Score divided by the fraction of articles published by the journal. That fraction is normalized so that the sum total of articles from all journals is 1.

• The mean Article Influence Score is 1.00. • A score greater than 1.00 indicates that each article in the journal has above-

average influence. • A score less than 1.00 indicates that each article in the journal has below-average

influence.More directly comparable to the Impact Factor.

SCImago Journal & Country Rank

• Metrics based on data from the database Scopus• Scopus: approximately indexes 15,000 journals (compared to Web

of Science: approx. 9,300 journals)• SCImago Journal Rank Indicator is similar to Eigenfactor• 27 major thematic categories (Subject Area)• 313 specific subject categories (Subject Category)• Comparison of up to four journals at the time• Map generator for visualization of fields

Source: Stock (2009). The inflation of impact factors of journals. ChemPhysChem, 10, p. 2193

Sources to look for journal impact indicators

Journal Citation Reports (data from ISI Web of Science)Network login is necessary if accessing it from outside of campus!

Journal Eigenfactor (data from ISI Web of Science)

Scopus Network login is necessary if accessing it from outside of campus!

SCImago Journal & Country Rank (data from Scopus)

Publish or Perish (free software)Uses Google Scholar to obtain raw citations and analyses the data.

Some aspects that influence journal evaluation

The impact factor – what are citations all about?

From: http://www.phdcomics.com/comics/archive.php?comicid=1108