IV Simpósio da AMP

Secção Regional do Centro da Ordem dos Médicos

7 de Novembro de 2015

Elsa Vieira de Sousa

Sociedade Portuguesa de ReumatologiaServiço de Reumatologia, Hospital de Santa Maria, CAML

Unidade de Investigação em Reumatologia, Instituto de Medicina Molecular, CAML

• Development of the Science Citation Index (SCI) and later the Journal Citation Reports (JCI).

• Impact factors became calculated yearly, starting from 1975, for those journals that are indexed in the Journal Citation Reports.

• Frequently used as a proxy for the relative importance of a journal within its field.

SCI® Journal Citation Reports®: a bibliometric analysis of science journals in the ISI® database. Philadelphia: Institute for Scientific Information, Inc.®,1993.

Garfield proposed that citations could be used as a quality filter

• The impact factor was devised by Eugene Garfield, the founder of the Institute for Scientific Information (ISI),nowadays Thomson Scientific.

In any given year, the impact factor of a journal is the averagenumber of citations received per paper, published in that journal,during the two preceding years.

2014 impact factor = A/B

A = 2014 cites of articles published in 2012 and 2013.B = the total number of "citable items" published in 2012 and 2013.

How is impact factor calculated?

Quantifying citations has become the post-publication counterpart to pre-publication peer review.

SCI® Journal Citation Reports®: a bibliometric analysis of science journals in the ISI® database. Philadelphia: Institute for Scientific Information, Inc.®,1993.

How useful is the impact factor?

• Provides librarians and researchers with a tool for the management of library journal collections

• Provides quantitative evidence for editors and publishers for positioning their journals in relation to the competition (in the same subject category)

• Serves advertisers interested in evaluating the potential of a specific journal.

• Helps authors and editors to identify the most appropriate, influential journals in which to publish.

http://wokinfo.com/products_tools/analytical/jcr/

The missuse of the impact factor

• Academic evaluation (institutions and individuals)

• To promote or hire, to decide salary and bonuses

• Evaluate research centres

• Sellect Rheumatology Excellence Centres, by the EuropeanLeague Against Rheumatism (EULAR)

Simons K, Science, 2008Rossner M, J Cell Biol, 2007

Impact factor limitations

• Citations vary according to:

– the type of article (reviews and methods)

– the categories (field dependent factors)

– the time after publication (first 2 years: 1-3% mathematicsand 5-8% biologic sciences)

. Bornmann, L., & Daniel, H. D, Journal of Documentation, 2008

Rossner M, J Cell Biol, 2007

Impact factor limitations

Rossner M, J Cell Biol, 2007

METHODOLOGY

• All published articles are considered for citations (numerator). Only those considered as “citable” are included in the denominator

• Articles are designated as primary, review, or “front matter” by hand by Thomson Scientific employees using various bibliographic criteria, such as keywords and number of references

• Some publishers negotiate with Thomson Scientific to change articles designations in their favor

• Citations to retracted articles are counted in the impact factor calculation

• The IF is calculated as a mean therefore is susceptible to outliers

Impact factor criticisms

• The outlier effect of a very cited article

• Difficulties to be consistently reproduced by independent audits

• Journal rankings constructed based solely on impact factors only moderately correlate with those compiled from the results of expert surveys

• It is important to note that impact factor is a journal metric and should not be used to assess individual researchers or institutions

• Editor policies

. Bornmann, L., & Daniel, H. D, Journal of Documentation, 2008

Rossner M, J Cell Biol, 2007

Acta Reumatológica Portuguesa

Case study

ARP milestones

FirstEdition

(1973)

Pubmed/Medline

(2006)

JournalCitationReports

1st rheumatologyibero-americanjournal

(2007)

Impact factor

1st Portuguese Medical Journal

(2010)

ARP online (2012)

The faces of ARP

The ARP impact factor conundrum

Artigos submetidos Percentagem total de rejeição

2013 259 26%

2014 332 38%

2015 329 (nov) 52%

60

108

90

6452

5165

0

20

40

60

80

100

120

2008 2009 2010 2011 2012 2013 2014

Publications/year

0,726

0,451

0,547

0,695

0,831

0,286

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

2009 2010 2011 2012 2013 2014

The ARP impact factor conundrum

Impact factor

38,33%

32,41%

51,11%

68,75%

53,85%

86,27%

83,08%

0,00%

20,00%

40,00%

60,00%

80,00%

100,00%

2008 2009 2010 2011 2012 2013 2014

English publications ratio

The ARP impact factor conundrum

12149

16089

19059

27472 27209

2062819131

0

5000

10000

15000

20000

25000

30000

2008 2009 2010 2011 2012 2013 2014

PubmedClicks

The impact factor effect

• On editor policies– Reviews by invitation

– Coercive citations

• On authors– Submit their papers to journals at the top of the impact factor ladder,

circulating progressively further down when they are rejected

– Scientists are increasingly desperate to publish in a few top journals and are wasting time and energy manipulating their manuscripts and courting editors.

– The objective presentation of work, the accessibility of articles and the quality of research itself are being compromised.

– Findings are sliced as thin as salami and submitted to different journals to produce more papers.

Lawrence P, Nature, 2003

• Howard Hughes Medical Institute – only a subset of publications chosen by a scientist for be review

• PLoS One• evaluates papers only for technical accuracy and not subjectively

for their potential impact on a field

Rethinking practices

• European Association of Science Editors (EASE) (2007)“Journal impact factors are used only—and cautiously—for measuring and comparing the influence of entire journals, but not for the assessment of single papers, and certainly not for the assessment of researchers or research programmes”

• International Council for Science (ICSU) Committee on Freedom and Responsibility in the Conduct of Science (CFRS) (2008) "statement on publication practices and indices and the role of peer review in research assessment", suggesting many possible solutions—e.g., considering a limit number of publications per year to be taken into consideration for each scientist, or even penalising scientists for an excessive number of publications per year—e.g., more than 20.

• German Research Foundation (2010) published new guidelines to evaluate only articles and no bibliometric information on candidates to be evaluated in all decisions concerning "performance-based funding allocations, postdoctoral qualifications, appointments, or reviewing funding proposals, [where] increasing importance has been given to numerical indicators such as the h-index and the impact factor". This decision follows similar ones of the National Science Foundation (US) and the Research Assessment Exercise (UK).

Rethinking practices

http://www.ease.org.uk/publications/impact-factor-statementhttp://www.icsu.org/publications/cfrs-statements/publication-practices-peer-review/statement-publication-practices-and-indices-and-the-

role-of-peer-review-in-research-assessment-july-2008http://www.dfg.de/en/service/press/press_releases/2010/pressemitteilung_nr_07/index.html

Lawrence P, Nature, 2003

Chief-Editors

João Eurico Fonseca (2006-2007)

Maria José Santos (2007-2008)

Helena Canhão (2009-2010)

Lúcia Costa (2011-2012)

Mónica Bogas/Joaquim Polido-Pereira (2013-2014)

Associated-EditorsAna Maria RodriguesAntónio AlbinoFernando PimentelFilipa MourãoFilipa RamosHelena CanhãoJoaquim Polido PereiraJoão Eurico FonsecaJosé António Pereira da SilvaJosé Carlos RomeuJosé Melo GomesLúcia CostaLuís GraçaMaria José LeandroMaria José SantosPedro MachadoSofia Ramiro

Technical Editor

João Cavaleiro

Acknowledments