Multiple Bibliometric

Indicators Approach to

Croatian Open Access

(OA) Journals

Franjo Pehar, Jadranka Stojanovski

Department of Information Sciences, University of Zadar

Libraries in the Digital Age 2014

“Better to understand a little than to

misunderstand a lot.”

Anonymous writer

Rise of metrics

● importance and usage of citation metrics has

increased substantially in recent years

● the most popular metric indicator is still JIF

● ...initially intended as a way to help libraries

to select the journals to be acquired…

● ...and to help academics to ‘filter content’

● used to measure academic achievement o at individual, project (grant) and institutional level

...

● academic career

● research project assessment

● institution ranking

● funding decisions

● …

● new situation has drawn significant criticism o practices that encourage conformity in research

o “salami-slicing” or incremental publishing

o unequal treatment of researchers from low-cited

disciplines

Eugene Garfield at LIDA 2004

“JIF is not

created for all the

purposes that

some people

want to use them

for. A lot of

people who are

using JIF are

doing it for

advertising

purposes.”

http://www.garfield.library.upenn.edu/

papers/medicalwritingv8(1)1999.html

“It is crucial to

know how

citations are

connected, how

they are related.”

“The real problem is when you

start using JIF as the

substitute for citation

analysis of individuals. I have

always said that you should

not use JIF to evaluate a

person or department. If I

manage to get my paper

published in Nature, does that

guarantee that it is a great

paper? Even if the paper is

never cited?”

http://researchimpact.files.wordpress.com/2011/10/research-impact-cartoon.jpg

What is a citation?

● simple line in the reference list (not) pointing

to the paper

● paper which act as a foundation for our own

work

● paper which help us to broaden our

knowledge

● tool to confirm that we prepared well prior to

our own research

● tool to flatter someone

● tool to “self-cite” at different levels ...

Citation as a metric tool

● many factors can influence the rate of citation o age of article, type of research conducted, the subject

area

o objectivity of citations? real benefit? (erroneous

references with hundreds, thousand citations)

● tools for evaluation and assessment are

evolving constantly, giving rise to new metrics o different aspects of citation and publication behaviour

Journals

● fee and free journals

● „OA journal” as a synonym for APC model

● journals? - or more portals

● very prominent journals = high JIF journals =

very for-profit journals

● „international journals” and „local journals”

● „regional journals”

Journals

● journal as an old concept from the printed

world

● journal as (still) main channel of scholarly

communication

● journals slowing down a possible

development of science (?), not

implementing ICT advantages and

possibilities…

● ...or implementing it at very slow pace

What’s about journals from the

scientific semi-periphery?

Narrower context

● Croatia o 4.5M population, 11k scholars, 7 universities, 25

research institutes, 25k papers per year and 1300

islands

● Croatian journals o challenges for small language and semi-peripheral

science context

Croatian journals in numbers

Academic career policies in Croatia

“papers must be published in journals with JIF above average for the specific

subject category”

“at least two-thirds of the papers must be published in the journals listed in SCIE”

“paper is rated regarding ratio of the JIF and median of the subject category, and

the minimal value must be 0.5

“the value of the qualitative criteria Q is calculated as a sum of the ratios of JIF

and median JIF of the subject category, multiplied by the factor Fd of the

candidate”

“Example: Journal of Engineering Mechanics in 2011 has JIF 0.99, subject

category is Engineering – Mechanical with median JIF 0.743 …”

“in the A1 category are the papers published in journals with JIF in Q1, Q2, i Q3

of the subject category ...in the A2 category are papers from the journals in

Q4…”

“in the A1 category are the papers published in journals indexed by Web of

Knowledge...if the JIF falls in the first 50% of the subject category...then the

paper is counted as two papers”

Hmmmm….

are Croatian journals important?

is it reasonable just to transfer the Croatian

journal funds to APC or fee based prominent

and popular “international” journals?

what justification do we have to publish

Croatian scholarly journals?

HRCAK - improved visibility,

readability...popularity...citations

HRCAK by disciplines

Objective of the study

● Which aspects of journal usage metrics differ

from citations metrics

● In particular, following issues were be

addressed: o Croatian OA journals usage

o Relationship of full-text paper downloads and

citation counts

o Comparison between full paper downloads and

citations counts relative to different subject areas

Materials and methods

● HRCAK usage statistics o Cover journal's web page visits

o Issue web page visits

o Article (bibliographic records) web page visits

o Full-text (PDF) downloads

o Unique visits from the same IP address during the

period of 30 minutes

...

● For the purpose of the study o Usage data for 115,458 full-text papers

(documents) acquired from 340 unique OA

journal titles March 2007 to the May 1st 2014

Article page visits

Full-text downloads

674 downloads

1293 visits

Research sample

● 47 OA journals (28,121 papers) available in

HRCAK repository and indexed by

o Web of Science™ Core Collection (Thomson

Reuters) 28,725 indexed papers

o Scopus (Elsevier) 28,953 indexed papers

Preprocessing challenges

● Assembling a single data source based on three

data sets

o lack of (P)ID’s for matching documents from

different data sets

● Titles were reduced to the same form

● Levenshtein distance algorithm to determine

similarity between paper titles (2 iterations) o 15,023 (53%) papers in Web of Science and 16,592

(57%) papers in Scopus matched with HRCAK titles

o 13,127 (ca. 46%) unique papers merged from all

three data sets

● 2 journals dropped-out

...

● Initial cleaning of usage data o validity, reliability, transparency ...

● Missing data o HRCAK page visit & download dates/years, more

reader demo-geographic information

● Matching/merging procedure o two journal titles completely lost

o further improvements possible > expected matching

rate ca. 85% (cat(ti ... jn … is … vl … pg)

Results

● download and citation frequencies (March

2007 - May 2014)

Matched

articles

HRCAK

visits

HRCAK

Downloads

Scopus

citations

WOS

citations

13,127 2,084,174 4,174,888 38,106 30,324

Visits vs Downloads

Article page visits: 1-250=84%;

Downloads: 1-250=59%, 251-500=22%

cor(log(1+HRVisits, HRDown),

method=”spearman”)

Possible reasons …

...

...

Scopus vs Web of Science citations

Scopus: 0=41%, 1-5=44%;

Web of Science: 0=47%, 1-5=42%

cor(log(1+HRDown, ScopusCit,

WoSCit), method=”spearman”)

Visits, downloads and citations

relative to different subject fields

Subject area No. journals % No. articles %

Biomedicine and

health

10 22.2% 5856 44.6%

Biotechnical

sciences

5 11.1% 1420 10.8%

Humanities 7 15.6% 527 4.0%

Sciences 8 17.8% 2037 15.5%

Social sciences 6 13.3% 1365 10.4%

Technical sciences 9 20.0% 1922 14.6%

Total 45* 100.0% 13127 100.0%

Median number of visits and

downloads & Mean number of citations

Median number of HRCAK visits and

downloads relative to six subject areas

Mean number of Scopus and Web of

Science citations relative to six

subject areas

Key results

● Different usage (reader) and impact (author)

characteristics

● Very strong, strong and low correlations o Scopus and WoS citation counts

o article page visits and downloads

o downloads and citation counts

● But, with significant differences between and

within six subject areas

Conclusion

● Important to have different (MULTIPLE) but

complementary measures for better

reflecting the multifaceted and

multidimensional character of scientific work

(output, impact, activities …)

Acknowledgment

• HRCAK team (University Computing Centre)

• Tomislav Jagušt (Faculty of Electrical

Engineering and Computing, University of

Zagreb)

Bieber’s response to yesterday's

accusation