uksg conference april 2010, edinburgh
DESCRIPTION
UKSG Conference April 2010, Edinburgh. The UKSG Usage Factor Project A Progress Report Richard Gedye and John Cox UKSG Conference April 2010, Edinburgh. UKSG Usage Factor Project. Brief background Issues addressed before data collection and analysis - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/1.jpg)
UKSG Conference April 2010, Edinburgh
![Page 2: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/2.jpg)
The UKSG Usage Factor ProjectA Progress Report
Richard Gedye and John Cox
UKSG Conference April 2010, Edinburgh
![Page 3: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/3.jpg)
UKSG Usage Factor Project 1. Brief background
2. Issues addressed before data collection and analysis
3. Collecting and analysing the data What data we have collected
Methodology
Issues and challenges
4. Anticipated issues that will need to be addressed
5. Next steps
![Page 4: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/4.jpg)
UKSG Usage Factor Project 1. Brief background
2. Issues addressed before data collection and analysis
3. Collecting and analysing the data What data we have collected
Methodology
Issues and challenges
4. Some early recommendations
5. Anticipated issues that will need to be addressed
6. Next steps
![Page 5: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/5.jpg)
The challenge…….
ISI's Impact Factor compensates for the fact that larger journals will tend to be cited more than smaller ones
Can we do something similar for usage?
In other words, should we seek to develop a “Usage Factor” as an additional measure of journal quality/value?
![Page 6: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/6.jpg)
For example…..
Usage Factor =
Total usage over period ‘x’ of articles published during period ‘y’
Total articles published during period ‘y’
![Page 7: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/7.jpg)
Usage factor advantages
Especially helpful for journals and fields not covered by ISI
Especially helpful for journals with high undergraduate or practitioner use
Especially helpful for journals publishing relatively few articles
Data available potentially sooner than with Impact Factors
![Page 8: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/8.jpg)
“Authors select journals that will give their articles prestige and reach. Impact Factor is a widely used surrogate for the former, while perceived circulation and readership reflect the latter. But usage is becoming more important as a measure of reach”
Carol Tenopir
![Page 9: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/9.jpg)
Modelling and Analysis Using Real Data
![Page 10: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/10.jpg)
Real journal usage data is currently being analysed by John and Laura Cox
Participating publishers:-
– American Chemical Society
– Emerald
– IOP
– Nature Publishing
– OUP
– Sage
– Springer
![Page 11: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/11.jpg)
UKSG Usage Factor Project 1. Brief background
2. Issues addressed before data collection and analysis
3. Collecting and analysing the data What data we have collected
Methodology
Issues and challenges
4. Anticipated issues that will need to be addressed
5. Next steps
![Page 12: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/12.jpg)
UKSG Usage Factor Project 1. Brief background
2. Issues addressed before data collection and analysis
3. Collecting and analysing the data What data we have collected
Methodology
Issues and challenges
4. Anticipated issues that will need to be addressed
5. Next steps
![Page 13: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/13.jpg)
Key data issues we have addressed
1. Consistency – numerator/denominator
2. Defining article usage year
3. Defining article publication date
4. Different usage patterns by subject
![Page 14: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/14.jpg)
Data issues we have addressed
1. Consistency
– Items in numerator must be in denominator
– Clear definition of qualifying “items”
Machine recognisable
Unambiguous
Solution? All items with a DOI?– This will include items such as editorial board listings,
calendars of events, sponsoring society announcements, etc.
![Page 15: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/15.jpg)
Other Possible Solutions
Rejected– Item must have references
– Item must not have an empty author field
– Item must be more than one page in length
Possible– Cross mapping items against one of the large and inclusive
A and I services or citation databases
– Examining article DTD tags
– Intelligent textmining
![Page 16: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/16.jpg)
Longer-term Solutions
Encourage publishers to:-
– Lodge more detailed article metadata with CrossRef
– Adopt the NLM DTD, use its article categories element, and make the results harvestable
![Page 17: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/17.jpg)
Key Data Issues
1. Consistency – numerator/denominator
2. Defining article usage year
3. Defining article publication date
4. Different usage patterns by subject
![Page 18: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/18.jpg)
Data issues we have addressed
2. Article usage year– Inter-journal comparisons can be distorted by different
patterns of article publication during the calendar year
– Usage in the first calendar “year” could be as little as one month and as much as 12 months
Solution– provide data about the first 12, 24, 36 months of usage
of articles published in each chosen calendar year rather than calendar year usage
![Page 19: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/19.jpg)
Key Data Issues
1. Consistency – numerator/denominator
2. Defining article usage year
3. Defining article publication date
4. Different usage patterns by subject
![Page 20: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/20.jpg)
Data issues we have addressed
3. Article publication date
– Early online version
– Final online version
– Printed issue publication date Some early or even “final” versions of articles
are published online many months (sometimes years) before the official publication date of the journal issue of which they are nominally a part.
Solution– Supply usage data at the article version level, showing
usage patterns of different versions separately
![Page 21: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/21.jpg)
Key Data Issues
1. Consistency – numerator/denominator
2. Defining article usage year
3. Defining article publication date
4. Different usage patterns by subject
![Page 22: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/22.jpg)
Data issues we have addressed
4. Potential differences by subject – Might usage patterns vary between subject areas?
– To find out, we needed to identify a third party schema which had classified by subject all journals participating in our project
Solution– Use the Dewey Decimal Codes (DDC) which the British
Library have assigned to all the journals for which they hold records (>20,000)
![Page 23: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/23.jpg)
With key data issues addressed, we developed a specification for a report via which participating publishers would deliver their usage data for analysis
![Page 24: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/24.jpg)
UKSG Usage Factor Project 1. Brief background
2. Issues addressed before data collection and analysis
3. Collecting and analysing the data What data we have collected
Methodology
Issues and challenges
4. Anticipated issues that will need to be addressed
5. Next steps
![Page 25: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/25.jpg)
UKSG Usage Factor Project 1. Brief background
2. Issues addressed before data collection and analysis
3. Collecting and analysing the data What data we have collected
Methodology
Issues and challenges
4. Anticipated issues that will need to be addressed
5. Next steps
![Page 26: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/26.jpg)
JUF variables to be tested All journal content (excluding standing matter)
Articles only:– Version of Record
– All versions of the article on the publishers’ platform
Differing publication periods – 1 or 2 years (2006-2009)
Differing usage periods:– Single year of usage from the online publication date
– Two years of usage from the online publication date
– Single year of usage from a year after the online publication date
– Two years of usage from a year after the online publication date
Samples of calendar year usage
![Page 27: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/27.jpg)
JUF variables to be tested – continued…Subject comparisons Broad subjects:
– Physical Sciences
– Medicine and Life Sciences
– Social Sciences
– Humanities
– Engineering
Narrow subjects– Business and Management
– Clinical Medicine
![Page 28: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/28.jpg)
Methodology
![Page 29: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/29.jpg)
The data
![Page 30: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/30.jpg)
The calculationJournal Usage Factor =
Total usage over period ‘x’ of items published online during period ‘y’Total items published online during period ‘y’
‘x’ is the usage period ‘y’ is the publication period
Create comparative subject data JUFs for each journal into seven spreadsheets (one per subject)
– All content JUFs
– Article only JUFs
– VoR only JUFs
– All version JUFs
![Page 31: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/31.jpg)
Comparing resultsVariable Article & Non-article content JUFs
Publ. Year
2006 2007 2008 2009 2006 - 2007 2007 - 2008 2008 - 2009
Usage Period
1-12 1-24 13-24
13-36
1-12 1-24 13-24
13-36
1-12 1-24 13-24
13-36
1-12 1-24 13-24
13-36
1-12 1-24 13-24
13-36
1-12 1-24 13-24
13-36
1-12 1-24 13-24
13-36
Journal Title
Journal A
12 2.3 15.4 15.2
Journal B
15 4.2 23.4 17.2
Variable Article only JUFs
Publ. Year
2006 2007 2008 2009 2006 - 2007 2007 - 2008 2008 - 2009
Usage Period
1-12 1-24 13-24
13-36
1-12 1-24 13-24
13-36
1-12 1-24 13-24
13-36
1-12 1-24 13-24
13-36
1-12 1-24 13-24
13-36
1-12 1-24 13-24
13-36
1-12 1-24 13-24
13-36
Journal Title
Journal A
16.3 4.1 18.4 18.9
Journal B
16.8 5.7 24.4 20.1
![Page 32: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/32.jpg)
Determine the best definitions for the calculation To include non-article content or not
To include versions of articles other than the VoR
Which definitions of ‘x’ and ‘y’ work best
Does calendar year create as meaningful data
Are the differences between subjects significant – do they need different definitions or calculations
What will be easiest for publishers
![Page 33: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/33.jpg)
UKSG Usage Factor Project 1. Brief background
2. Issues addressed before data collection and analysis
3. Collecting and analysing the data What data we have collected
Methodology
Issues and challenges
4. Anticipated issues that will need to be addressed
5. Next steps
![Page 34: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/34.jpg)
UKSG Usage Factor Project 1. Brief background
2. Issues addressed before data collection and analysis
3. Collecting and analysing the data What data we have collected
Methodology
Issues and challenges
4. Anticipated issues that will need to be addressed
![Page 35: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/35.jpg)
Issues to address in next phase of project
Detecting and deterring gaming
Differences between disciplines and journal types
What about print usage
What about offline usage
How to integrate usage data when journal content hosted on multiple sites
Responding to technological innovations
![Page 36: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/36.jpg)
Responding to technological innovations
Prefetching to local cache(E.g. PubGet, WebFeat)
– Need to establish list of user-agent names
– Then ignore prefetch requests and count only those with a “304” response
![Page 37: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/37.jpg)
Responding to technological innovations
Bulk downloading to local hard disk
(E.g. Quosa, PubGet bulk download plug-in)
– If specifically requested (e.g. Quosa), these should ideally be counted but considered separately
– We are still considering ways to address the automated downloading of articles to hard disk
![Page 38: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/38.jpg)
UKSG Usage Factor Project 1. Brief background
2. Issues addressed before data collection and analysis
3. Collecting and analysing the data What data we have collected
Methodology
Issues and challenges
4. Anticipated issues that will need to be addressed
5. Next steps
![Page 39: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/39.jpg)
UKSG Usage Factor Project 1. Brief background
2. Issues addressed before data collection and analysis
3. Collecting and analysing the data What data we have collected
Methodology
Issues and challenges
4. Anticipated issues that will need to be addressed
5. Next steps
![Page 40: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/40.jpg)
Next steps Submission to UKSG of final report from John and Laura
Cox – end of July 2010
This report will:-
– Outline the various metrics assessed
– Recommend which of them prove consistent and robust enough to be adopted for scaled up onward monitoring
– Suggest any ways in which data providers might amend the way they capture, structure, label, and maintain their data which would make the measurement of Usage Factors:-
Easier More reliable
– Propose ways to audit Usage Factors for accuracy
![Page 41: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/41.jpg)
UKSG Research Committee will consider the report and decide whether it justifies seeking funding for a further (third) phase for the Project
![Page 42: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/42.jpg)
UKSG Usage Factor Phase 3
Scaled up testing of candidate metric(s) recommended in Cox report
Address outstanding issues revealed during the course of the project so far
In collaboration with data suppliers, develop agreed standards and templates which, going forward, will streamline the process of data collection and analysis
More detailed practical recommendations for a cost-effective infrastructure to manage the Usage Factor process.
![Page 43: UKSG Conference April 2010, Edinburgh](https://reader035.vdocuments.site/reader035/viewer/2022062422/56813e89550346895da8c6cf/html5/thumbnails/43.jpg)
UKSG Usage Factor Project
Many thanks to the sponsors of this latest phase:- GOLD
SILVER– ALPSP
– American Chemical Society
– STM
– Nature Publishing Group
– Springer