Making a difference: The impact of scholarly publication

in addictionJudit H. Ward, Director of Information Services,

Paul Candon, Managing Editor, Journal of Studies on Alcohol and Drugs,

Center of Alcohol Studies, Rutgers, the State University of New Jersey

ELISAD Conference, Budapest, October 8-10, 2009

Agenda

• Background• Performance indicators• The Project• Conclusions• Q&A

Disclosure of bias and authorship: Prepared by the Center of Alcohol Studies Library staff for the Journal of Studies on Alcohol and Drugs, both part of the Center of Alcohol Studies, Rutgers University

Background: Center of Alcohol Studies

• Interdisciplinary research center at Rutgers University (New Jersey) since 1962

• Research in • Biological sciences (biochemistry and

neuropharmacology) • Psychology (clinical and experimental) • Sociology

• Education and Training• Continuing professional education• Summer school of alcohol studies

• More at: alcoholstudies.rutgers.edu

Background: Information Services

• Library and Archives• one of the world’s largest and most comprehensive

English-language collections on alcohol abuse• mission: to organize, preserve, and disseminate

information for scholars and clinicians in the field

• Research done by• librarian, library school student, library school

professor, editor, researcher, author, publisher

Background: Journal of Studies on Alcohol and Drugs

• The oldest substance-abuse journal published in the United States

• Multidisciplinary: epidemiological, psychological, psychiatric, biological, medical, social, and legal aspects of alcohol and drug use, abuse, and dependence

• Online access to the journal's archives dating to 1975 for subscribers

• One of the most cost-effective of all the psychology and substance-abuse journals

• Name change: QJSA 1940, JSA 1975, JSAD 2007Please see poster for more information

The Project: Significance

Interdisciplinary research Googlization

e-only journals BUDGET CRISIS

international collaboration preprints

NEW MODELS OF SCHOLARLY COMMUNICATION

NEW ACCESS TO CURRENT RESEARCHworking papers

open access new technologiesdiscussion forums

NEW FORMS OF SCHOLARLY PRODUCTS

Facebook Twitter Blogs

The Project: Impact• Why now? Another research with unimportant

discoveries? Mr. Turner and the “merformance”test?

• Direct impact– Unveil JSAD trends, corroborate anecdotal evidence,

verify assumptions, fix errors– Understand how audience (scholars, librarians,

competitors) views JSAD via available tools and indicators

– Learning opportunity in bibliometrics, research methods, group decision making, collaboration, and communication

• Long-term impact– Still a work in progress

Performance Indicators: Terminology

• Working definition: quantitative and qualitative statistical information used to assess performance

• Metrics-based research includes– Bibliometrics: "the application of mathematics and

statistical methods to books and other media of communication“ coined by Alan Pritchard in Journal of Documentation (1969) 25(4):348-349

– Scientometrics: science in general, not just publications– Informetrics: includes all information objects– Webmetrics, cybermetrics: focuses on web relationships

• Literature review nearly impossible without focus– Each slide has abundant literature

Performance Indicators: Users

• Scholars• AU: monitor own performance in research context,

peers’ performance, tenure and reappointment• JN: what to read and where to publish, invitations

(reviewer, editor, board member)

• University administrators• Performance of unit or individual scholar/educator• Hiring and tenure decisions• Budget and financial support: fund allocations,

policies

Performance Indicators: Librarians

• Evaluating JN performance for collection development: subscriptions, cancellations, weeding, formats

• Quantitative analysis: basis and/or justification for decisions

• Organization of information for patrons• Research assistance for patrons: scope, impact

factor, trends, based on history of science• LIS research: complementing qualitative methods

with quantitative (similarly to social sciences)• WorldCat for books & journals, Ulrich’s Periodical

Directory

Performance Indicators: Publishers

• Monitor trends and make decisions• Identify patterns, structure and dynamics,

predictions• Interested in

• authors and institutions who contribute• the most highly cited authors and articles• types of articles drive the impact factor• percentage of papers with no citations• geographical distribution• impact of special issues and supplements• expected citation rates• ranking among peer journals

The Project: Research Design• Participants: Judit Ward, Paul Candon, James

Cox, Molly Stewart, Nicole Frisone • Goal: Identify citation patterns of articles published

in JSAD over time• Methods: Using a variety of citation analysis

software, compute and analyze citations for particular periods

• Communication: shared workspace, F2F, informal

• Data storage– Electronic: refdesk computer, RefShare– Print: at refdesk in folders organized by topics

The Project: Questions • MEASURES?

– Citation analysis software: Which ones? How many? Proprietary vs. free? Benchmarking? Growth? Productivity? Mapping?

• CRITERIA?– AU? TI? Citation? Depth? Cited articles vs citing articles? Co-

citation?– Publishing period: X-year cycle? Selected years? Selected

issues? Selected articles? Number of issues or articles in period– CAS authors only? Institution? RU? CAS? Collaborations?

• DATA?– Top 10? Top 25? 0 citation?– Subject categories: DB driven? Compare apples and oranges?– Textual analysis? Frequency? Classification?

• ETC?

Citation Maps

Cited

Cited

Cited

Cited

Cited

Cited

ARTICLEARTICLE

CitingCiting

CitingCiting

Impact: 1-gen Forward Citation

Web of Science, (Thomson Reuters, 2009)

More Impact: 2-gen Forward Citation

Web of Science, (Thomson Reuters, 2009)

1-gen Backward/Forward Citation

Web of Science, (Thomson Reuters, 2009)

The Project: Breakdown• JSA, JSAD representation and metrics• A few representative articles from JSA or

JSAD first authored by CAS researchers• One highly-cited article by a CAS author as

benchmark• One outstanding CAS author as benchmark• Compare findings• Assess results in the context of addictions

publications

The Project: Workflow

Institute for Scientific Information (ISI)

• Founded by Eugene Garfield in 1960

• Market leader • Most comprehensive

data• Widely used &

accepted• Web of Knowledge• Web of Science• Current Contents

• Citation Indexes (Humanities, Social Sciences, Science)

• Journal Citation Report

• ScienceWatch• Impact Factor• Eigenfactor• ISI Highly Cited• ResearcherID

Journal Citation Reports®

• journal performance metrics for over 7,500 scholarly and technical journals

• statistical information based on citation data

• includes virtually all areas of science, technology, and social sciences

• helps assess influence in the research community

• shows impact in the context of journal and field

Ranking of Addiction Journals

2008 Journal Citation Reports® (Thomson Reuters, 2009)

Impact FactorNumber of citations received in current year by

papers published in the journal in the previous two years

DIVIDED BYNumber of papers published in the journal in the

previous two years

2008 Journal Citation Reports® (Thomson Reuters, 2009)

Eigenfactor™ and Article Influence™

• Eigenfactor™: the number of times articles from the journal published in the past five years have been cited in the JCR year

• EF Category : Psychology– 18th out of 62

• JCR Category:Substance Abuse -4th out of 23

• Article Influence™ Score: relative importance of the journal on a per-article basis.

• =Eigenfactor Score divided by the fraction of articles published by the journal (sum total of articles from all journals is 1)

• mean Article Influence Score =1.00.

• score less than 1.00 => each article in the journal has below-average influence

• score greater than 1.00 =>each article in the journal has above-average influence

use citation data from JCR to assess and track the influence of a journal in relation to other journals

Free and searchable at www.eigenfactor.org

Cost effectiveness2nd out of 23

Impact Factor - Eigenfactor• Both: a ratio of number of citations to total

number of articles• Eigenfactor Score: citations to journals in

both sciences and social sciences • Eigenfactor Score: eliminates self-citations

(references from one article in a journal to another article from the same journal is discounted)

• Eigenfactor Score: weights each reference according to the amount of time researchers spend reading the journal

Example: JSA scores and trends

JSAD in Journal Citation Reports®

2008 Journal Citation Reports® (Thomson Reuters, 2009)

Journal Citation Reports®

Citations from the journal (per cited year)

Citations to the journal (per cited year)

distribution by cited year of citations to items published in the journal

distribution by cited year of citations from current-year items in the journal

2008 Journal Citation Reports® (Thomson Reuters, 2009)

IncitesThe top 20 journals in Social Sciences

ScienceWatchExample: Top papers/journals in Special Topics Underage/College Drinking

Free resource! See also Top institutions, Top authors, etc.

Example: AU Performance

Source: ISI (BUT 248 listed in ISIHighlyCited) instead of 114 results

ISIHighlyCited.comSM

• free database of researchers• the most highly cited within each category for

the period 1981-1999• top individual researchers in each of 21

subject categories– biographical information: appointments, honors,

memberships– Complete listing of publications with links to

bibliographic information or full text• membership is an honor

H-index (Hirsch index, Hirsch number)

based on two factors: cited papers and the number of their citations in other publications

“A scientist has index h if h of [his/her] Np papers have at least h citations each, and the other (Np - h) papers have at most h

citations each” (Hirsch, 2005, PNAS, 102, p. 16569)a scholar with an index of 27 has published 27 papers each of

which has been cited by others at least 27 times

• problem: each DB uses it own data

Publish or Perish: AU

Publish or Perish: AU

Publish or Perish: JSA

Publish or Perish: JSAD

SCOPUS: JN Profile

SCOPUS: AU Profile

SCOPUS: AU

SCIMago: Data from SCOPUS

SCIMago: Data from SCOPUS

Going with the flowPresence on the Internet

Performance 2.0?

Summary: Types of problemsJournal• Name change JSA>JSAD, records not updated (JCR, PubMed, WorldCat,

Scopus, etc.)

Authors • No name authority file, or ignored• Name variations (Helene Raskin White: H. White, H.R. White)• Transliteration (Vaschillo, Vashchillo)

Institution• Center of Alcohol Studies within Rutgers University• Name variations: Rutgers, the State University of New Jersey

Articles ( see example on next slide)• White, H.R. and Labouvie, E.W. 1989. Towards the Assessment of

Adolescent Problem Drinking. Journal of Studies on Alcohol 50:30-37.

Example: the most cited article

Conclusions

• No single source is sufficient• Context of the field: multidisciplinary,

evolving• Comparison, corroboration • Multiple perspectives of diverse users• Soliciting feedback from patrons,

subscribers• Practice critical thinking in evaluation