comparing scientific performance across disciplines: methodological and conceptual challenges
TRANSCRIPT
Comparing scientific performance across disciplines: Methodological and conceptual challengesLudo WaltmanCentre for Science and Technology Studies, Leiden University
7th International Conference on Information Technologies and Information SocietyNovember 5, 2015
2
THE ranking
3
Shanghai ranking
4
Website University of Ljubljana
5
Impact factors
6
Centre for Science and Technology Studies (CWTS)• Research center at Leiden
University focusing on science and technology studies
• History of more than 25 years in bibliometric and scientometric research
7
Key challenges in measuring scientific performance1. Defining scientific performance2. Designing sensible metrics3. Correcting for field differences4. Quantifying productivity5. Assessing validity of metrics6. Balancing simplicity, accuracy, and robustness to
gaming
What are Slovenia’s research strengths?
8
Challenge 1: Defining scientific performance
9
Indicators of scientific performance• Publications:
– Total– Per faculty– Per student– Interdisciplinary– International collaboration– Nature and Science
• Citations:– Total– Per publication– Per faculty– Highly cited researchers
• Reputation survey• Others:
– Nobel Prizes/Field Medals– PhDs awarded– PhDs awarded per faculty– Post-doc positions– Research income
10
What do we mean by scientific performance?Size-dependent concept of scientific performance:• Overall contribution of a research unit to science• Total number of ‘performance points’ (e.g.,
publications, citations, expert recommendations, awards)
Size-independent concept of scientific performance:• Contribution of a research unit to science relative to
available resources• Number of ‘performance points’ divided by
available resources (e.g., number of faculty, research budget)
11
Indicators of scientific performance• Publications:
– Total– Per faculty– Per student– Interdisciplinary– International collaboration– Nature and Science
• Citations:– Total– Per publication– Per faculty– Highly cited researchers
• Reputation survey• Others:
– Nobel Prizes/Field Medals– PhDs awarded– PhDs awarded per faculty– Post-doc positions– Research income
Size-dependent indicators Size-independent indicators
12
Mixing up different concepts of scientific performance• Shanghai, THE, QS, and US News use composite
indicators• These composite indicators combine size-
dependent and size-independent indicators
It is unclear which concept of scientific performance is measured by Shanghai, THE, QS,
and US News
13
Challenge 2: Designing sensible metrics
Definition of the h-index
14
A scientist has index h if h of his papers have at least h citations each and the other papers have
at most h citations each
15
Arbitrariness of the h-index
Consistency requirement
16
If two scientists achieve the same performance improvement, their ranking relative to each other
should remain unchanged
17
Inconsistency of the h-index
18
Newly proposed metric: Relative Citation Ratio
This metric has the peculiar property
that receiving additional citations
may cause the metric to decrease
rather than increase
19
Challenge 3: Correcting for field differences
20
Differences in citation density between fields
0 1 2 3 4 5 6 7 8 9 100
5
10
15
20
25
30
35
publication age in years
aver
age
num
ber
of c
itatio
ns p
er p
ublic
atio
n
biochemistry & molecular biologycardiac & cardiovascular systemschemistry, analyticalsurgeryeconomicsphysics, appliedmathematics
21
Differences in citation density between fields
22
Field-normalized indicators
Elsevier SciVal
THE ranking
23
Differences in citation density within fields (clinical neurology)
24
About 4000 fields of science in the CWTS Leiden Ranking
Social sciences and
humanities
Biomedical and health sciences
Life and earth sciences
Physical sciences
and engineering
Mathematics and computer science
25
CWTS Leiden Ranking
26
CWTS Leiden Ranking
27
SNIP: Source normalized impact per paper
28
Differences in citation density within fields (clinical neurology)
29
Differences in reference density within fields (clinical neurology)
30
Citation density vs. reference density (clinical neurology)
Citation density Reference density
31
SNIP: Source normalized impact per paper• Impact per paper (IPP) of journal X in 2013:
• Source normalized impact per paper (SNIP) of journal X in 2013:
C11CIPP P P
i
PR1
SNIP
C
1 i i
Number of publications in journal X in 2010–2012
Number of citations in 2013 to publications in journal X in 2010–2012
Number of references in citing publication i to publications in 2010–2012
Be careful in using SNIP in the humanities!
32
Challenge 4: Quantifying productivity
33
Quantifying productivity
• Same resources as Univ. B• P = 1000• P(top 10%) = 200• PP(top 10%) = 20%
• Same resources as Univ. A• P = 2000• P(top 10%) = 300• PP(top 10%) = 15%
Univ. A Univ. B
• Taking into account that both universities have the same resources, it is clear that university B has performed better
• However, according to the PP(top 10%) indicator, university A has performed better
34
Challenge 5: Assessing validity of metrics
35
Correlating metrics with peer review
Correlating journal impact metrics with Norwegian expert
judgments
Correlating citation metrics with F1000 recommendations (post publication peer review)
36
Correlating metrics with peer review
Correlating various metrics with UK REF expert judgments
37
Difficulties of using peer review to validate metrics• Biases in peer review, e.g.:
– Gender– Interdisciplinarity– Conservatism
• Influence of metrics on peer judgment– Did REF expert panels truly make their decisions without taking
into account any metrics?
• Limited precision of peer review outcomes doesn’t allow accurate comparisons of metrics:– Norwegian expert judgments: 3 classes– F1000 recommendations: 3 classes– UK REF expert judgments: 5 classes
38
Challenge 6: Balancing simplicity, accuracy, and robustnessto gaming
39
40
Suspicious editorials: Time trend
• At least 50 citations to publications in own journal in past 2 years
• At least 75% journal self citations
41
Suspicious editorials: Effect on IF
42
Coercive citing
“you cite Leukemia (once in 42 references). Consequently, we kindly ask you to add
references of articles published in Leukemia to your present article”
43
Balancing simplicity, accuracy, and robustness to gaming
Simplicity
Accuracy
Robustness to gaming
IF vs. SNIP?
Exclude citations from
editorials?
Exclude journal self-citations?
44
How to deal with these challenges?
45
46
Leiden Manifesto
1. Quantitative evaluation should support qualitative, expert assessment
2. Measure performance against the research missions of the institution, group or researcher
3. Protect excellence in locally relevant research4. Keep data collection and analytical processes
open, transparent and simple5. Allow those evaluated to verify data and analysis
47
Leiden Manifesto
6. Account for variation by field in publication and citation practices
7. Base assessment of individual researchers on a qualitative judgement of their portfolio
8. Avoid misplaced concreteness and false precision9. Recognize the systemic effects of assessment
and indicators10.Scrutinize indicators regularly and update them
48
Slovenia’s research strengths
49
Top 5 Slovenian research strengths
Social sciences and
humanities
Biomedical and health sciences
Life and earth sciences
Physical sciences
and engineering
Mathematics and computer science
50
Top 5 Slovenian research strengths
Keywords:Journals:
Institute:
acrylic acid; ethylene terephthalate; polypropylenesurface & coatings technology; j. of applied polymer science; plasma processes & polymersjozef stefan institute
Keywords:Journals:Institute:
cholesteric liquid crystal; holographic polymer; nanoparticlemolecular crystals & liquid crystals; physical review e; liquid crystalsjozef stefan institute
Keywords:Journals:Institute:
evolution; dilemma game; altruismj. of econ. behavior & organization; games & econ. behavior; j. of theoretical biologyuniv. of maribor
Keywords:Journals:Institute:
acidic medium; mild steel corrosion; hydrochloric acid solutioncorrosion science; int. j. of electrochemical science; corrosionjozef stefan institute
Keywords:Journals:Institute:
wood density; wood property; treeeur. j. of wood & wood products; holzforschung; forest products journaljozef stefan institute; nat. inst. of biology; univ. of ljubljana
51
Thank you for your attention!