measure up! the extent author-level bibliometric indicators are appropriate measures of individual...

Measure up!The extent author-level bibliometric indicators are

appropriate measures of individual researcher performance

Lorna WildgaardRoyal School of Library and Information Science

Faculty of the Humanities, CPH U27th November 2015

OVERVIEW1. Background

2. Characteristics and effects of ALI (Papers 1, 2, 4)

3. The appropriateness of ALI across different disciplines and different academic seniorities (Papers 3, 5, 6, 7)

4. Concept definintion in the construction of ALI(Chapter 6 & Appendix B, http://tinyurl.com/nj4mvca)

5. Conclusions

Background Preliminary Analyses Empirical Analyses Validation Study Conclusion

CORRECTION


“ Bibliometrics is the application of variousstatistical analyses to study patterns of authorship, publication, and literature use.” (Lancaster, 1977).

“[…] to analyse the structure of science, measure science, and indicate the production, citations and collaboration of researchers, institutions and countries.” (De Bellis 2014; Pritchard 1969).

4

1980s 1990s

2000s

2010s1970s1960s1950s

de Solla Price growth model

Namilov interdisciplinary

approach to indexing scientific literature

Merton Social structure of scientific communication

Haitun; Yablonsky distribution of citations

Garfield SCI & IF

h-index


5

Managerial

Teaching

Outreach

Societal

Technical Skills

Funding/Grants

Productivity

Quality

Relevance

Visibility

Reputation

Scientific

Counting, adding,

dividing, multiplying an authors’ publications & citations


DEFINTITIONS OF AU, P & C IN ALIAuthorship, publication and citation is different from discipline to discipline, from time to time and location to location and has implications in indicator development (Bošnjak and Marušić, 2012)

What constitutes a publication [author and citation] should be clearly defined to ensure representative operationalization in the indicator and the extraction of meaningful relationships (Wouters 1999; Skupin 2009; Colledge 2012)


EVALUATION

OUTPUT

DATA

ALI MODEL

MOTIVATION TO PUBLISH

MOTIVATION TO CITE

INTERPRETATION OF ALI

LEGITIMATE LINK TO REAL WORLD

PERFORMANCE OF RESEARCHER IN SYSTEM REDEFINES WHAT SUCCESS IS

EXOGENOUS VARIABLES

WHAT IS MISSING

INDEXING POLICY


THE PROBLEM

1. What are the characterisitics of ALI of academic performance?

2. To what extent are ALI appropriate in the evaluation of researchers from different disciplines and different academic seniorities?

3. To what extent are the concepts being measured defined in the construction of ALI?


RESEARCH QUESTIONS

9

ALI GIVE A SNAP-SHOT OF SELF IMAGE AND CORE PERSONALITY TRAITS

COMPARING RESEARCHERS CAN EXPOSE THE INDIVIDUAL

INDIVIDUALS USE ANY DATA TO INCREASE ALI SCORES, TO INCREASE THEIR SUBJECT VALIDITY & SELF-WORTH

ALI BRING OBJECTIVITY TO THE EVALUATION & REDUCE GENDER; CULTURAL AND RACIAL BIAS

DOCUMENTING BEING OUT-PERFORMED IS DETRIMENTAL TO A RESEARCHER’S SELF-DEFINITION

ALI DO NOT ADD CONTEXT BUT CAN ADD REDUNDANT INFORMATION

SUCCESS IS DEFINED AS WELL IN THE SYSTEM

EXTERNAL CHARACTERISTICS OF ALI


INTERNAL CHARACTERISTICS OF ALI

ALI are designed to measure particular aspects of the effect of a researcher’s work – over time, to field, as quality, ranking all or selected works, co-authorship etc.

Judgements based on ALI can lead to assumptions about the productivity and impact of a researcher, which can be unsubstantiated, and affect the psychological character of the individual.

ALI have to be methodologically sound

PRELIMINARY ANALYSIS: RESULTS


FAMILY CHARACTERISTICS


METHOD: Compute simple indicators on publication and citation data of 750 researchers (585 m, 165w)

4 disciplines, 5 academic senioritiesCorrelation (Kendall’s Tau) as measure of association between seniority and ALI, discipline by discipline ALSCAL IBM SPSS v.19

ENDPOINTSuitability of indicators in different disciplines and seniorities


FAMILY CHARACTERISTICS

DISCIPLINE CENTRAL INDICATOR S-STRESS % VARIANCEASTRONOMY hg 0.37 25ENVIRONMENTAL SCIENCE h, h2 0.37 24

PHILOSOPHY IQP 0.38 47PUBLIC HEALTH g 0.49 38

METHOD: Publication and citation data for 512 researchers22,143 papers, 423, 371 citations (WOS)52,227 papers, 746,985 citations (GS)For each researcher 17 ALI calculated

Agreement: determined by matching rank positions Variability of rank position determined by standard deviation of the difference in scholar rank position, calculated from the matched pairs

END POINTS: agreement in ranking between WOS & GS

RANKING CHARACTERISTICS



hg produced highest level of agreement in rankings in WOS & GS, 0.8-0.9 tau across disciplines and seniorities (Patel 2013)

hg fulfills its potential as a ranking indicator (Alonso et al 2010) yet this tells us nothing about the excellence of the researcher.

h and g cannot be rationally combined to produce indications of research excellence (Franceschini & Maisano 2011).

Indicator rankings inform of the researcher’s visibility in the citation index not in their academic community (Bar-Ilan 2008; De Battisti 2012; Farhadi et al 2013).


RANKING CHARACTERISTICS

METHOD: Same data as previous studiesExplorative statistical study of distribution; two step cluster; ordinal regression: odds ratiosIBM SPSS v.22

ENDPOINT: Identify disciplinary & seniority appropriate indicators


GROUPING CHARACTERISTICS

CLUSTER 1below IQR, high nnc, sc, low

P & C. Adjusting for academic age mg, m.quotient,

currency of cites improved scores.

CLUSTER 2Median IQR, high scores on

collaboration indicators

CLUSTER 3Top IQR,

highest scores normalized for field, one SIG

paper

CLUSTER 4Extreme outliers,

highest C, cited quickly, highest htype scores

ASTRO HEALTHPHILENVIRO

h2 SUM PP TOP PROP

Q2 e

GROUPING CHARACTERISTICS

Ordinal regression: Academic age is statistically significant for cluster placement Within cluster rank position determined by the ratio P:h

1. Simple ALI are complex!2. Basic descriptive statistics can be more informative3. There are families of indicators4. ALI scores are highly influenced by the database,

academic age and the fit of the data to the model5. ALI are more appropriate in some disciplines than

others 6. Different ALI produce stable rankings within and across

databases.7. ALI can be easily manipulated8. Scores can be due to chance rather than practical

importance.

EMPIRICAL ANALYSIS: CONCLUSIONS


Knowing all these practical challenges, what properties should a well constructed indicator contain?

METHOD:Based on papers 1-7Validate ALI using Gingras’ criteriaAnalyze operationalization of AU, P, C in ALI

ENDPOINT:Recommendation of a set of disciplinary dependent indicators of scientific publication output


VALIDITY ANALYSIS

THE 3 CRITERIA OF PROFESSOR YVES GINGRAS

1. Adequacy of the indicator to the object it measures -quantity of items a researcher publishes is an indicator of production

2. Sensitivity to the intrinsic inertia of the object -changing values in the time interval correspond to the speed and direction of change in the object

3. Homogeneity of the dimensions of the indicator -h-index is a heterogeneous indicator: it mixes quantity of papers published with citation counts


10

28

HOMOGENEITY

ADEQUACY INERTIABackground Preliminary Analyses Empirical Analyses Validation Study Conclusion

17 did not fulfill any of the criteria

n=69 ALI

4 1

720

OPERATIONALIZATION OF AU, P, C”THE LOGIC GRID”

METHOD:• Extract definitions and hypotheses in ALI• Matrix of definitions of AU, P, C & aim of ALI

AIM:Identify in a simple way the properties of ALI


HIRSCH QUALITY IMPACT USE OVER TIME IDEA TRANSFER

CITATION

6 6 6 5 2PERFORMANCE EFFECT INFLUENCE POPULARITY REWARD NO DEF.

2 1 1 1 1 20

PUBLISHED SCIENTIST AUTHOR AWARD WINNER

PUBLISHED & CITED SENIORITY NO DEF.

AUTHOR 12 11 10 7 5 2 5

PAPER IN WOS PAPER IN OTHER INDEX PAPER OBJECTS WITH

CITATION EXPRESSION

PUBLICATIONS 30 15 6 1 1

EXCELLENCE EFFECT INDEPENDENCE COMPARE RANK CURRENCY

MEASURE

9 7 7 6 6 4

GROWTH QUALITY DISTRIBUTION QUANTITY & QUALITY CAREER DURABILITY

3 3 2 2 1 1

LOGIC GRID

http://tinyurl.com/ofm7h8s

• ALI are designed for specific author and publication types

• Majority of ALI designed for papers in WoS • Some indicators are purely theoretical• Inherent bias in all ALI is that they are

designed for researcher profiles befitting the hard sciences

• Lack of definitions = methodological deficiency

If developers cannot define the variables they are measuring, how can we?


1. TransparencyKnow which data is used to compute the

indicatorUnderstand the math and the inferences

2. DemographicsBe aware if demographics affect the ALI scores

3. MotiveALI must fit the objectives of the evaluation

4. DiversityChoose ALI that fit the discipline

5. OpenessMake the limitations of ALI explicit, use

supplementary ALI

PROPERTIES OF A WELL CONSTRUCTED ALI


RQ 1: What are the characterisitics of ALI of academic performance?

ALI are complex in character. They use different:

• disciplinary perspectives, • objectives, • operationalization of variables, • requirements to data, and • mathematical models that favour different disciplines

and seniorities.


CONCLUSIONS

one size does NOT fit all.

RQ 2: To what extent are ALI appropriate in the evaluation of researchers from different disciplines and different academic seniorities?

• No seniority ALI were identified

• There is a disconnection between the performance of researcher on the CV and in ALI.

• Some ALI are more appropriate in some disciplines than others as grouping and ranking indicators or as indicators that isolate unique information


CONCLUSIONS

RQ 3: To what extent are the concepts being measured defined in the construction of ALI?

• Many indicators are poorly or only partially defined

• Decisions based on these ALI could turn out to be less effective than expected

• If we do not know how the developer has done the measurement, we cannot repeat it, reflect on the meaning associated with the concept and compare results to previous findings


CONCLUSIONS

THANK YOU!!ACUMEN partners: especially Judit Bar-Ilan, Frank van der Most & Paul Wouters.

My supervisors: Jesper Schneider, Peter Ingwersen & Birger Larsen

PhD/Masters group: Jesper, Niels-Peder, Maria, Ole, Rikke, Sara, Sille, Lisa Sutherland & BADASS

Researcher/teacher support: Johanne Maibohm, Susanne Acevedo, Pia Dithmar & Ragnhild Riis.

My boys:Kim, Zander & Balthazar

My opponents: Henk MoedRodrigo Costas & Jeppe Nicolaisen

measure up! the extent author-level bibliometric indicators are appropriate measures of individual...

Leadership & Management