Finding Insights in ALMSfor Research Evaluation

Richard Cave, @richardcave

I.T. Director, PLOS

Charleston Conference 2013

Article-Level Metrics at PLOS

• Article-Level Metrics (ALM) allow the comprehensive analysis of the post-publication activity around a paper, employing usage stats, citations and altmetrics.http://article-level-metrics.plos.org/alm-info/

• Started in 2009 at PLOS• Lots of data, tracking over a dozen sources

2

ALMs provide the flexibility to look at a wide range of numbers

3

Authors want to see the impact of their article

4

What about funders and institutions?

• Funders want to monitor the research that they support– Wellcome Trust spends approximately £650 million on

research per year. What’s the impact?– Médecins Sans Frontières – how do they know the

reach of their article?

• Institutions want to see how their researchers are making a change– Stanford University – do librarians know what research

was published by Stanford researchers?

5

Numbers are GREAT

but what’s the impact of the research?!?

6

How easy it for institutions and funders to track research impact?

• It’s not easy!• Aggregate data for all articles? Hard to do• Follow tweets? Not likely• News/blog stories? Hard to find correct sources

7

8

ALM Reports

almreports.plos.org

Launched June 18

Allows researchers, institutions & funders to: • Create a report of the ALMs for a

set of PLOS articles • View a summary of the metrics

along and data visualizations.

But more important, provides a way to provide context of ALMs with search

Search based on: • Keyword• Author name & country • Institutional affiliation• Publication date• Subject areas• Funder

9

10

For researchers at your institution, can show impact for all of their articles

For institutions and funders - Get specific!

(author_affiliate:"Stanford University") AND (financial_disclosure:"National Institutes of Health" )

(author_affiliate:"Stanford University") AND (financial_disclosure:"Wellcome Trust")

(author_affiliate:"Stanford University") AND (financial_disclosure:"National Cancer Institute")

(author_affiliate:"Stanford University") AND (financial_disclosure:"National Cancer Institute" AND financial_disclosure:"Doris Duke Foundation")

11

Page views and citations over time

Standford University + National Cancer InstituteTotal usage includes page views and downloads from PLOS and PMC. Bubble size correlates with Scopus citations and bubble color with the PLOS journal.

12

Page views and Mendeley bookmarks over time

Standford University + National Cancer InstituteTotal usage includes page views and downloads from PLOS and PMC. Bubble size correlates with Mendeley bookmarks and bubble color with the PLOS journal.

13

Rectangle size correlates with total views (HTML and PDF) from PLOS website. Color intensity correlates with number of Scopus citations.

Article usage and citations by subject category

Standford University + National Cancer Institute

Research collaborationsStandford University + National Cancer Institute

15

Wellcome Trust evaluation

Adam Dinsmore at Wellcome Trust:• Identified and exported the entire corpus of PLOS articles

associated with the Wellcome• Analysed the ALMs associated with them and benchmarked

against the sector average• Produced a consumer report

Findings:• Wellcome Trust associated PLOS articles are viewed, saved and cited

more often than the average for the biomedical research sector• They are discussed by the blogosphere more often than the sector

averagehttp://article-level-metrics.plos.org/files/2013/10/Dinsmore.pptx

16

For Publishers• Improve Business Intelligence

– Optimize publicity efforts with more targeted campaigns (Call For Papers, conference presence, etc.)

– Launch new collections of papers in areas of interest – Find new fields growing at the intersections between communities of

interest

• Deepen Editorial Capacities– Direct editorial resources to promote new research of interest in

growing fields– Identify promotional opportunities for papers and authors (writing

press releases, front matter content, editorial overviews, etc.)– Uncover potential new authors

• Deliver a Richer Product– Maximize front matter visibility – Create more targeted editorial overviews – Enhance press reviews and promotional content for high-impact

papers17

But beware of dragons

• Data issues = Garbage in, garbage out• Author names• Funder names

• “Welcome Trust” vs. “Wellcome Trust”• Institutional names

• A dozen ways to enter “University of California, Berkeley”

• Will be helped in the future by:• ORCID • FundRef

18

ALM info

• General information on ALM

http://article-level-metrics.plos.org/• ALM Reports

http://almreports.plos.org/– Future: ALM Reports will be open source but will have

limitations (no search functionality)

• ALM Application is Open Source (Ruby on Rails)https://github.com/articlemetrics/alm – Recently launched at Copernicus Publications– OJS has a beta trial

19