Evaluation Insights to Key Processes of Digital Repositories

Konstantinos Koumoutsos, Angelos Mitrelis Giannis TsakonasLibrary Information Center, University of Patras, Greece.

Libraries In the Digital Age 2010, May 24-28, 2010, Zadar, Croatia

Digital Repositories

• A familiar stranger:– Important channels for disseminating scientific/academic

productivity.– Important means for preserving scientific/academic outputs.– Representing different types of communities (institutional/

thematic).– Wide-spread systems, but with high degree of uniqueness.– Reflecting different modes of policies and interactions.

2

Digital Repositories’ Evaluation

• Institutional Repositories:– Social aspects• impact of scientific documents, motives of self-archiving activity,

relationships with other types of digital repositories, etc.– Technical aspects• effectiveness/efficiency measures of submission, interfaces, etc.

• DSpace* in particular:– Comparative evaluation of system features– Evaluation of end user performance– Administration/configuration easiness

* Used in this study

3

is Research

• Part of a wider evaluation initiative for the ‘Nemertes’ IR.• Emphasizes in key processes held inside the repository:– Submission (users as self-submitters).– Retrieval (users as searchers).– Editing (librarians, moderators, administrators).

4

Context

• ‘Nemertes’, the University of Patras’ IR.– Developed by Library and Information Center.– Since 2006 on a DSpace installation (v 1.4).– Link: http://nemertes.lis.upatras.gr

• Collections:– Self-submission.– ‘eses and Dissertations of University of Patras’

(primary collection, also on OPAC, printed/digital format)– ‘Technical Reports’– ‘Journals/Proceedings Publications of the LIC personnel’– ‘Faculty Members Publications’

(pre/post-prints of the faculty members of the University) 5

Methodology ~ Overview

Submission process: a questionnaire survey addressed to people who had earlier submitted in the IR.

Retrieval process / Interfaces: a HCI-based user study.

Editing processes: interviews with librarians.

6

Methodology ~ Some Details

• uestionnaire survey:– Previously submitted in the IR.– From 1129 registered, 126 participated.– Online, 25 questions, scale from 1 (-) to 5 (+).

• HCI-based user study:– 24 graduate students, Department of Electrical Engineering,

HCI background.– Based on usability heuristic evaluation principles (10 principles,

4 levels of problem significance).• Interviews:– 5 librarians supporting the service.– Semi-structured, approx. 19 minutes duration.

7

Insights to Submission

• Reported: Overall satisfaction with the process.• Fairly satisfied with help facilities and submission wizard.• Satisfied with the support of the librarians.• Very satisfied with the self-submission and the

bilingualism of the system.

8

Further Insights to Submission

• Satisfied with copyright (now: institutional license/plus an option for a Creative Commons one).– Yet 15.1 % do not know or do not answer.

• Satisfied with exclusion period (publication embargo)– Alternatives: most: up to 2 years / a few: up to 5 years.

• Fairly satisfied with access policy (now: free access to all)– Alternatives: most: controlled access to everyone / a few: closed

to everyone.

9

Insights to Retrieval Interfaces

Principle Unique problems

Number of Reports (by Severity)

Number of Reports (by Severity)

Number of Reports (by Severity)

Minor Important SeriousVisibility of system status 4 2 4 3Match between system and the real world 17 10 29 20User control and freedom 6 3 9 3Consistency and standards 12 5 24 8Error prevention 4 1 14 2Recognition rather than recall 3 1 11 4Flexibility and efficiency of use 9 9 13 5Aesthetic and minimalist design 20 0 74 18Help users recognize, diagnose, & recover from errors 4 0 11 7Help and documentation 5 5 6 2

36 (11.88%)

195 (64.36%)

72 (23.76)

10

Further Insights to Interfaces

11

x16

x6

x9

x8x3

Further Insights to Retrieval

12

x16

x1

Insights to Editing

• e editing process is long and time consuming. – Yet minimizes the possibility of errors.

• Retrospective editing of metadata and corrections to files is considered essential.

• Submitters’ keywords are considered invaluable.• Reporting problems in users’ navigation in the physical

space to accomplish a submission.• Receiving complaints for the completion of the license.

13

Further Insights to Editing

• Usual problems:– e absence of abstracts and keywords in parallel language.– e errors in bibliographic descriptions (capitals, mixed

keyboard languages, ‘greeklish’).– e uploading of damaged or non compliant files.

• Auto-suggestion fields are recommended.• Faculties and Departments should assist in promoting

‘Nemertes’.• Library should be proactive.

14

A Wide View of Findings

• Submitters seem very positive. Librarians and ‘HCI users’ were strict.

• Search functionalities need revision.• Interface features need clarity and consistency.• Editing needs revision to enhance efficiency.

15

Response

• Migration to new DSpace version.• Resolve problems in information retrieval.• Apply add-ons that assist submitters and librarians.– Secure correct entry of metadata.

• Careful consideration of labels and terms.• Exploit submitters’ keywords.• Design a new theme and graphical elements.

16

Conclusions

• An obvious conclusion: more research. – More views are needed.– Each collection poses new challenges.

• e three processes are linked. – Each process has an effect on the other.– e physical aspect of the service affects the digital.

17

e End

• ank you for your attention.• uestions?

Addendum:• ‘Nemertes’: http://nemertes.lis.upatras.gr/dspace/handle/123456789/3069• ‘E-LIS’: http://eprints.rclis.org/18502/• Contact: Giannis Tsakonas, john@lis.upatras.gr

18