Xiaodan Zhu and Peter Turney National Research Council Canada

Daniel Lemire TELUQ, Université du Québec Montréal

Andre Vellino School of Information Studies, University of Ottawa, Ottawa

Measuring Academic Influence: Not All Citations Are Equal

Overview �  Some background in Citation Analysis �  What we tried to do and why �  How we did it �  What the results were �  What the implications are

What is Citation Analysis Citation analysis refers to the collection of methods for measuring the importance of scholars, journals and institutions by counting citations in a graph of references in the published literature.

…

…

… …

…

…

Why Do Citation Analysis?

�  Reason # 1: Because it generates measurable quantities!

“Since we can’t really measure what interests us, we begin to be interested in what we can measure”

Joel Westheimer Professor of Education

University of Ottawa

Uses for Citation Measures �  For Readers

� To evaluate the quality of articles / journals

�  For Universities �  To evaluate the productivity of academics � To help in tenure and promotion decisions

�  For Journals � To attract authors to publish

�  For Libraries � To make collections / acquisition decisions � To make automated recommendations to users

How Are Citations Counted? �  Add 1 for every new occurrence of a cited article �  Sum the results �  Average per article & / or Count Total # of citations Problems �  Self citations! �  No measure of quality of citing source �  May be skewed by a small number of highly cited items �  Easy to “game” by tricking Google Scholar

�  viz. Ike Inktare h-index = 94 – Einstein h-index = 84

h-index �  Jorge Hirsch (PNAS, 2005) defined the h-index:

� Attempts to measure both the productivity and impact of the author’s published work

� An author has index h if h of their N papers have at least h citations each, and the other (N − h) papers have at most h citations each.

Some Criticisms of the h-index �  The h-index does not account for the number of authors or the order of

the authors of a paper. �  Cannot use the h-index to compare authors in different fields �  Young researchers with as yet short careers are at a built-in disadvantage

over older researchers �  Constrained by the total number of publications

�  10 papers each w/ 100 citations each = 10 papers w/ 10 citation each

“[h-index] captures a small amount of information about the distribution of a scientist's citations [and] loses crucial information that is essential for the assessment of research.” 

Adler, R., Ewing, J. Taylor, P. Citation statistics. A report from the International Mathematical Union. http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf

Journal Impact factor (IF) �  Invented by Eugene Garfield in 1955 to identify journals for

Science Citation Index �  Definition:

Total Citations (2 preceding years )

Total Articles (2 preceding years ) = JIF

i.e. the impact factor of a journal is the average number of citations to those papers that were published during the two preceding years ¨  e.g. the number of times articles published in 2001 and 2002

were cited by indexed journals during 2003 / the total number of items published in 2001 and 2002

Some Criticisms of Impact Factor �  Letters or editorials in some journals (e.g. Nature) are often cited

(and counted) in “Total Citations” (numerator) but not in “Total Articles”

�  2-year window not applicable in many fields (e.g. in Math 90% of citations fall outside the 2-year window)

�  IF varies considerably across disciplines (Math has an average of 0.9 citation per article, Life Sciences have an average of 6.2)

“Using the impact factor alone to judge a journal is like using weight alone to judge a person's health.” 

Adler, R., Ewing, J. Taylor, P. Citation statistics. A report from the International Mathematical Union. http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf

What We Did and Why

�  As early as 1965 Garfield identified 15 different reasons for citing �  giving credit for related work �  correcting a work �  criticizing previous work

�  Many attempts since to categorize citations

One Big Assumption All citations should count equally!

Citation Typing Ontology (CiTO)

Here are first 21 of the 91 citation types in CiTO

http://imageweb.zoo.ox.ac.uk/pub/2008/plospaper/latest/#refs

Example of semantically annotated article using CiTO:

Our Objective �  Solve a binary classification problem:

Given a Paper-Reference (P-R) pair, does P-R belong to the class “R is highly influential for P” or not.

Our Method �  Apply Machine Learning methods to train a computer to

recognize “Highly Influential Reference” from examples

Step 1 – Data Collection

We believe that most papers are based on 1, 2, 3 or 4 essential references. By an essential reference, we mean a reference that was highly influential or inspirational for the core ideas in your paper; that is, a reference that inspired or strongly influenced your new algorithm, your experimental design, or your choice of a research problem. Other references merely support the work.

We asked for �  Title of your paper (research papers only; no surveys) �  The essential references does your paper build?

We got �  100 papers �  322 “influential” references

�  i.e. 3.2 “influential references” per article �  Each paper

�  Contained ~ 31 references in the References section �  Cited ~ 54 references in the body of the paper

�  i.e. each reverence was cited an average of 1.7 times per paper

The Problem �  The 100 papers yield 3143 paper-reference pairs �  The authors have selected ~320 paper-reference pairs

�  Algorithmically: to accurately select those 320 from the 3142

Paper – Reference Analysis �  OpenNLP used to detect sentence boundaries and tokenize. �  ParsCit to parse the papers.

�  ParsCit is an open-source package for parsing references and document structure in scientific papers.

�  Regular expressions to capture citation occurrences in paper bodies that were not detected by ParsCit.

Characteristics of Corpus

We Looked at 5 Classes of Features 1.  Count-based features 2.  Similarity-based features 3.  Context-based features 4.  Position-based features 5.  Miscellaneous features

Count Based Features �  Total number of times a paper is referenced in the citing paper �  The number of different sections in which a given reference appears �  Number of times a paper is referenced in the

�  “Related” section �  “Introduction” section �  “Core” sections (all sections excluding “Related”, “Introduction”,

“Acknowledgements”, “Conclusion” and “Future Work” �  The number of different sections in which a reference appears

Content-Similarity Based Features Citing article Referenced articles

Title-Title

Title-Abstract

Title-Conclusion

Title-Introduction

Title-Core

Citing Context �  When an article is cited, the linguistic context in which the

article is cited is considered as saying something about the cited article.

e.g. “Like Moravcsik and Murugesan (1975), we are concerned about the side effects of counting insignificant references”

Context-Similarity Based Features Citing Article

Title Abstract Introduction Conclusion

Other Context Based Features �  Authors explicitly mentioned in citation context? �  Citation alone [4] or with others [3,4,5] �  If “with others” is it first? (e.g. “[3]” is first in “[3,4,5]”) Using pre-defined word-lists, is the lexical content of a citation �  “relevant” [likewise, influential, inspiring useful….] �  “new” [recently, latest, current, improved…] �  “extreme” [greatly, intensely, acutely, almighty, awfully] �  “comparative” [easy, easier, easiest, strong, stronger…]

Lexical Context Features Using a lexicon of 114,271 words obtained from the General Inquirer Lexicon (11,788 words) extended w/ Wordnet + Turney and Littman Algorithm, �  Count the number of words labeled

�  “Strong” �  “Positive” �  “Evaluative”

Also, sentiment analysis with a different lexicon gave us �  Presence / absence of “Emotion” (Joy, Sadness, Anger, Fear, etc.) �  “Positive” / “Negative”

Position Based Features Where does the citation occur? �  Citation appears at the beginning of a sentence? (Y/ N) �  Citation appears at the end of a sentence? (Y/N) �  Where are the sentence(s) in which the citation(s) occur(s)

e.g. �  0 (First sentence) to 1 (Last sentence) �  distance from the mean of occurrences of all citations

Count Based Features

Similarity Based Features

Context Based Features

Position Based Features

Misc. Features

Top 7 Features: 4 “counts”, 3 “similarity”

Counts in Paper Counts in Sections Counts in Core Section Title-Abstract Similarity Counts in Intro Section Title-Core Similarity Title-Intro Similarity

Conventional Measures on Citation Graph

…

…

… …

…

…

C R 1

Influence Primed Measures

…

…

… …

…

…

C R X

where X = (number of times C cites R)2

hip-index �  Each occurrence of a citation of paper R by paper C = 1 �  hip-index (h-influence-primed) index for an author is the

largest number h such that at least h of the author's papers have an influence-primed citation count of at least h.

Examples hip-index = 5 h-index = 2 cited 3 times by C1 = 9 cited 2 times by C2 = 4 cited 2 times by C3 = 4 cited 2 times by C4 = 4 R3 – cited 3 times by C5 = 9 R4 – cited 3 times by C6 = 9 R5 – cited 3 times by C7 = 9 R6 – cited 2 times by C8 = 4 R7 – cited 1 times by C9 = 1

13

8

9

9

9 4

1

hip-index = 3 h-index = 2 cited 2 times by C1 = 4 cited 1 times by C2 = 1 cited 2 times by C3 = 4 cited 1 times by C4 = 1 R3 – cited 2 times by C5 = 4 R4 – cited 1 times by C6 = 1 R5 – cited 1 times by C7 = 1 R6 – cited 1 times by C8 = 1 R7 – cited 1 times by C9 = 1

5

5

4

1

1 1

1

R1

R2

R1

R2

Using hip-index to Predict ACM Fellows �  Used the citation network constructed from �  ~ 20,000 papers in the Association for Computational Linguistics

Anthology

�  Calculated the h-index of ACL Fellows �  Calculated the hip-index of ACL Fellows �  Compared the precision of h-index and hip-index

�  the number of ACL Fellows in the top N divided by N

1/2 2/3

1/4

2/6

3/10 3/9

4/11 4/10 5/11

5/12

Conclusions �  We can throw away h-index and Impact Factor etc. completely

OR we can try to improve them by counting citations more relevantly

�  A measure of academic influence for a citation is possible and �  It is easy to compute to a first approximation – merely count

their frequency �  Apply the influence-primed weights on citation graphs to

compute �  Influence-primed Impact Factor, g-index etc.

Thanks!