berlin10 beyond the impact factor

Beyond the Impact Factor: Why the Thomson-Reuters impact factor has to be replaced

Tom Olijhoek

SURF NL

AcknowledgementsPaul Wouters Leiden University Jelle Wicherts Tilburg UniversityBjörn Brembs Regensburg

This work is licensed under a Creative Commons Attribution 3.0 Unported License

http://creativecommons.org/licenses/by/3.0/deed.en_US



Beyond the impact factor Tom Olijhoek SURF-NL

You are free to:

Copy, share, adapt, or re-mix;

Blog, live-blog, or post video of;

This presentation. Provided that:

You attribute the work to its author and respect the rights and licenses associated with its components.

Slide Concept by Cameron Neylon, who has waived all copyright and related or neighbouring rights. This slide only ccZero.Social Media Icons adapted with permission from originals by Christopher Ross. Original images are available under GPL at;http://www.thisismyurl.com/free-downloads/15-free-speech-bubble-icons-for-popular-websites

http://creativecommons.org/publicdomain/zero/1.0/

http://www.thisismyurl.com/free-downloads/15-free-speech-bubble-icons-for-popular-websites


The Thomson Reuters impact factor is used to assess the quality of a journal

The TR impact factor CORRELATES VERY WELL with the perceived quality of a journal

SO WHAT IS WRONG WITH IT?


MANY THINGS ARE WRONG WITH THE IMPACT FACTOR

Fortunately Open Access enables other methods for Quality Assessment

But…..the Impact Factor is an obstacle for Open Access

To get Open Access we need to get rid of the Impact Factor

For that we need an attitude change

For that we need commitment of scientist communities

In all parts of the world


Especially in the developing world

Participation of scientists in the developing word will make the difference

To participate on an equal basis in the making of science

To profit on an equal basis of the fruits of science

Science is the motor of economic development

OPEN ACCESS IS THE KEY TO EDUCATION, INNOVATION, ECONOMIC DEVELOPMENT AND

PROSPERITY EVERYWHERE ALSO IN THE DEVELOPING WORLD


Lars Bjørnshauge Quote from talk at the PKP conference September 2011, Berlin:

The push for researchers from [developing countries] and continents to publish in high impact factor journals has decisive influence on the subject of their research and much more so is a big obstacle for open access publishing.


2006 and 20072008

IF=5Articles published in 06/07

were cited an average of 5 times in 08.

citations articles

The Impact FactorIntroduced in 1960’s by Eugene Garfield: ISI


The impact factor

Nothing wrong with using citations quality criteria for articles and authors

Using average citations for the average article as quality indicator for a journal

That is where things go wrong

You can NOT draw conclusions on INDIVIDUAL article qualities based on the AVERAGE quality of

ALL ARTICLES in a journal


The USE of the impact factor for ASSESSING SCIENTISTS is

obstructing the move towards Open Access

Weak correlation of individual article citation rate with journal IF

But scientists are judged on NUMBER of publications in HIGH IMPACT

JOURNALS

Most scientists do not publish in OA journals for one reason: because it

could hamper their careers

Our management discourages us from supporting new open access journals due to their low, or unassigned, impact factor MalariaWorld survey 2012


Seglen PO (1997): Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314(7079):497 (15 February)

http://www.bmj.com/cgi/content/full/314/7079/497

Weak correlation of individual article citation rate with journal IF

http://www.bmj.com/cgi/content/full/314/7079/497


other problems with the Impact Factor

as an Indicator for Quality

Björn Brembs http://www.slideshare.net/brembs/limited-access-is-a-symptom-not-the-disease

Negotiable

Irreproducible

Can be gamed

PLoS Medicine, IF 2-11 (8.4)(The PLoS Medicine Editors (2006) The Impact Factor Game. PLoS Med 3(6): e291. http://www.plosmedicine.org/article/info:doi/10.1371%2Fjournal.pmed.0030291)

Rossner M, van Epps H, Hill E (2007): Show me the data. The Journal of Cell Biology, Vol. 179, No. 6, 1091-1092 http://jcb.rupress.org/cgi/content/full/179/6/1091

Journals exert pressure to get cited themselves oftenScientists „ask“ to be citedFake authors

http://www.slideshare.net/brembs/limited-access-is-a-symptom-not-the-disease

http://www.slideshare.net/brembs/limited-access-is-a-symptom-not-the-disease

http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0030291




Current Biology IF from 7 to 11 in 2003Bought by Cell Press (Elsevier) in 2001…

NEGOTIABLE: number of articles which are cited can be adjusted



What is obvious from this equation is that the impact factor depends crucially on which article types Thomson Scientific deems as “citable”—the fewer, the better (i.e., the lower the denominator, the higher the impact factor).


NEGOTIABLE



Methodological problems with the Impact Factor

• Retraction rates: high • Subjective journal rank: very

high• Quality of individual articles: low • Citations: low• Expert opinion: low • Methodological standards: low• Replicability: none

Björn Brembs and Marcus Munafò http://bit.ly/WNzA1Z

Correlation of IF with:


Two More Problems

€30,000-130,000/year subscription ratesCovers ~11,500 journals (Scopus covers ~16,500)

The Impact Factor is commercially produced

50,000 employees

US$600million profit/quarter

Thomson family owns 53%

€30,000-130,000/year subscription rates

The TR-Impact Factor underscores research topics from the South by design


OPEN ACCESS ENABLES THE DEVELOPMENT OF NEW WAYS OF QUALITY ASSESSMENT

While Open Access is hindered by the TR Impact Factor new methods made possible by Open

Access can replace it!

Measure Impact beyond mere citation analysis

Measure Impact beyond scientific impact


‘New’ Definition of Scientific Impact


We Need a Change in Attitude “When I light my candle from yours, I gain from you without subtracting from you. That’s what

sharing knowledge is like”. Peter Suber

Open Accessgetting new ideas by

sharing

Collaboration

Publish for impact

Focus on quality

Toll Access

fear of losing ideas

Competition

Publish or perish

Focus on quantity

We shouldn’t be counting the beans but instead taste themThe proof is in the pudding


Open Access opens Science for All

Science is the motor for economic

development

Research is the key to fighting

disease

Participation of scientists from

Africa, Asia and Latin America is

necessary for success

We need commitmentOpen Access is crucial for scientists in the global south

conference The Hague 25 oct 2012as long as scientific output remains behind walls of paid content, no possibility for a dialogue will exist


Specs for a global system

Open Access can contribute to that, provided that the existing dominant features of the existing system:

Citation counts and the JIF measures of impact (inadequate, insufficient and subject to gaming)

will be replaced by

measures that much better reflects the impact of research not only on research itself, but on innovation, health, wealth and societies

(altmetrics).

Luckily these are as well requirements of a successful breakthrough of OA in the North.

The Hague Oct 25th 2012

Lars BjørnshaugeSPARC Europe – www.sparceurope.org


ultimately scientists need to realize that they hold the power in their own hands

TAKE ACTION


When did scientists start agreeing with this slave-type of agreement with publishing houses? How could this nonsense have started? We inherited this sick system, but that does not mean we should allow it to continue

MalariaWorld Survey on Open Access 2012


A new Journal Impact Factor?Do we really need it?

In an Open Access worldWe only need:

• article level metrics to assess the impact of articles AND scientists

• a quality indicator (seal?) but NOT impact factor to assess the quality of journals

In an Open Access world It does not matter much where something is

published, more important is the quality of the individual

articlesHOW GOOD VERSUS HOW MUCH AND WHERE


The Relevance Index

Use of new metrics to assess the impact of scientific works in all areas not only

science but including innovation, health, wealth and societies


The Reputation Index

Use of the Relevance Index to assess the reputation of authors / scientists


We need a Quality Indicatorfor (0pen access) journals

FOR NEW JOURNALS NONE OF EXISTING METRICS WORK:NOT ENOUGH CITATIONS, NO REPUTATION YET

SO WE HAD TO DEVELOP SOMETHING NEW

AUTHORS NEED TO BE ABLE TO CHOOSE GOOD QUALITY JOURNALS

WHY?

New quality assessment methods in science

Articles

• Citations

• NEW• Relevance Index• Multi-level tools• Total Impact• Altmetric

explorer

Journals

• Impact Factor• H-Index• Citation based

• NEW• A-Vector• Based on quality

of peer review & quality of editorial board

Authors

• Citation index

• NEW• Reputation

Index• Subjective• Based on more

than citations


A-VECTOR: INTRODUCING A NEW FORM OF JOURNAL LEVEL METRICS

Results of the Rotterdam colloquium held on 22-23 October 2012

Quality of editorial board

• Citation index• Reputation• Collaboration• Reference density• More indicators

Quality of peer-review

• “Transparency” indicators• Criteria used by reviewers• Duration of review process• Post-publication comments• Openness about• submission and Rejection rates• potential conflicts of interest• Aims, scopes and expected

readership• Reviewer’s comments and editorial

correspondence ( published alongside papers

• More indicators


PREDICTION OF JOURNAL QUALITY BY A-VECTOR USING EDITORIAL BOARD MEMBER QUALITIES AS INDICATORS

CONCLUSION Quality of editorial

board members can be used to judge the (potential) quality of Open Access journals


• Prediction of Impact Factor based on TCS (total citation score), MCS (mean citation score), H-index of Editorial

Board Members• Performance test: prediction of IF for established journals

with error of 1.5 points (95% confidence)

• Requires some refinement, but can be applied to young OA journals

PREDICTION OF JOURNAL QUALITY BY A-VECTOR USING PEER-REVIEW TRANSPARENCY FACTORS AS INDICATORS 100 journals / 221 authors

CONCLUSION:

authors’ assessments of the quality of the peer-review of accepted papers can be predicted by using a set of 15 indicators of transparency for the peer-review process of journals


17 established journals: score = 45.4, SD =11.9 12 OA established journals: score = 54.7, SD = 7.7 13 OA Predatory journals: score = 34.1, SD = 6.7

EXAMPLES:Recent Scientific Research score 24

PLoSONE score 67.7 Malaria Journal score 50MalariaWorld Journal score 47


Item/criterion [authors] Convergence (η2)

Item-rest correl.

1. Aims, scope, and expected readership of the journal are clearly specified on the journal’s website .866 .457

2. Types of submissions that are deemed appropriate for the journal are explicated on the website

.899 .587

3. Criteria used by reviewers to rate submissions are specified on the website .692 .699

4. The website indicates whether all submissions are sent out for review and who will make final decisions about them (e.g., editor, associate/action editor)

.821 .660

5. The website provides timely updates of the status of submissions during the peer-review process (e.g., under review)

.749 .723


Reliability

Journal q1 q2 q3 q4 q5 q6

1 5 5 5 5 5 5

1 5 5 1 4 3 2

1 5 4 3 1 5 5

1 4 2 1 4 4 2

2 4 4 5 3 5 3

2 5 5 5 5 5 3

2 5 5 4 3 4 3

2 5 5 3 3 5 3

3 5 5 4 4 4 2

Across items: ReliabilityAcross items: ReliabilityA

cross raters: Convergence

Across raters: C

onvergence

Convergence across raters (13 journals): .838Reliability across items .900

YOU CAN NOW USE A-VECTOR YOURSELVES TO PREDICT JOURNAL QUALITY USING THE NIEW TRANSPARENCY FACTORS


Go to the link :

http://tinyurl.com/8br9m8w

And complete the survey

YOU CAN NOW USE A-VECTOR YOURSELVES TO PREDICT JOURNAL QUALITY USING THE NIEW TRANSPARENCY FACTORS


Applying the indicator

The journals we would like you to assess are as follows:

Journal no. 40

Journal title: Regional Studies

Journal URL http://www.tandfonline.com/toc/cres20/current

Journal no. 46

Journal title: The Internet Journal of Psychiatry

Journal URL http://www.ispub.com/journal/the-internet-journal-of-psychiatry/

Journal no. 8

Journal title: BMC Neuroscience

Journal URL http://www.biomedcentral.com/bmcneurosci/

The indicator can be found here: tinyurl.com/8br9m8w

berlin10 beyond the impact factor

Documents