Eric T. Meyer & Kathryn EcclesOxford Internet Institute, University of Oxford

JISC6th September 2011

How to measure success: Understanding and monitoring

impact

@etmeyer#tidsr#oess

What is impact and why consider it?

What do we mean by impact?

• Reaching intended audience• Reaching new audiences• Attracting users• Attracting new usage• Enabling new research questions• Enabling new approaches to education• Embedding a resource within a community• Embedding a resource within an educational environment

Measuring usage and impact

What can you measure?

• Users

• Types of use

• Awareness

• Citation practices

• Marketing strategies

• Embedding

• JISC funded project

• July 2008-April 2009

• Looked at five specific JISC-funded resources

• Designed to test the TIDSR methods and review them for the TIDSR toolkit

TIDSR: The first usage and impact study

Methods

Quantitative methods

• Webometrics

• Web Analytics

• Log file analysis

• Scientometrics / bibliometrics

• Content Analysis

Qualitative methods

• Interviews

• Focus groups

• User feedback

• Referrer analysis

• Content Analysis

Project 1 – Online Historical Population Reports (OHPR/Histpop)

Survey: Low Awareness

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

88%

63%

39%

63%69%

7%

27%

35%

28% 22%

3% 7%

18%

7% 7%2% 2%

7%1% 2%

Use it regularly or frequently

Use it on occasion

Have seen it, but don't use it

I haven't heard of it

Survey: High Importance to Users

HistPop BOPCRIS BL News BL SoundsMed Backfiles0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

71%

51% 49%

36%

61%

29%

35% 34%

40%

33%

79%

69%72%

60%

76%

96%

84%

90%86%

82%

Important to my research

Important to my teaching

Important to field

Would recommend

Log Files: Non UK Activity

Log File Analysis

Top Search Phrases: Histpop

Histpop: User Communities

Perception: Specific niche community Well known by target audience Transforming access and usage patterns

User surveys: Embedded in educational resources Enhanced access to primary sources

▪ ‘Histpop made it possible to do a completely different project’

Continuing education, online resources, non-traditional learners

Project 2 – British Library 19th Century Newspapers

Citation Habits

HistPop

BOPCRIS

BL News

BL Sounds

Med Back-files

9%

36%

53%

38%

43%

55%

36%

38%

50%

48%

36%

21%

6%

13%

10%

0%

7%

2%

0%

0%

Original version Original + URL Online version Other

0%10%20%30%40%50%

46%

29%

35%

20%

43%

Have you ever published a piece based on your work in this collection?

If so, how did you cite the collection?

Webometric results

• Highest numbers for original British Library resource (analogue)• 19th Century British Library Newspapers registers strong links for a project page• Note: Importance of comparator sites when using webometrics

Blog Evidence

Project 3 – British Library Archival Sound Recordings

Interviews, Group Interviews, Focus Groups

Time intensive, but productive if you are careful about what you ask!

Different stakeholders: Project team: Positive view of the work

only Broader stakeholders: While the digital

project was good, it also introduced tensions in the broader setting of the library

New kinds of serendipity, wide range of users

News

Engagement officer

7%

12%

16%

17%

31%

34%

45%

47%

51%

54%

58%

62%

71%

77%

83%

13%

11%

9%

30%

18%

29%

32%

38%

27%

24%

32%

36%

56%

50%

48%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Cornell Animal Sounds

Sciper

Histpop

Chronicling America

Fine Rolls

Wellcome Medical Journal Backfiles

Historical Directories

Internet Lib of Early Journals

BOPCRIS

Archival Sounds

Imperial War Museum

Old Bailey Online

British Periodicals

British Library Newspapers

House of Commons Parliamentary Papers

Non-UK Awareness

UK Awareness

Awareness of Resource by Country

How did you find this resource?

0%5%

10%15%20%25%30%35%40%45%50%

HistPopBOPCRISBL NewsBL SoundsMed Back-files

http://microsites.oii.ox.ac.uk/tidsr/

Some recommendations:

• Think about impact from the beginning

• Set your goals – what steps will you need to take?

• Identify connections:

• Which resources do you see as successful in terms of audience and impact?

• Is your resource connected to a community of resources? How can you use these connections?

Eric T. Meyereric.meyer@oii.ox.ac.uk

http://www.oii.ox.ac.uk/people/?id=120

Kathryn Eccleskathryn.eccles@oii.ox.ac.uk

http://www.oii.ox.ac.uk/people/?id=138

Oxford e-Social Science Project

Project work funded by:

Slides at: http://www.slideshare.net/etmeyer