repositories and digital preservation

A centre of expertise in digital information management

www.ukoln.ac.uk

UKOLN is supported by:

Repositories and Digital Preservation

Michael DayResearch and Development Team Leader

UKOLN, University of Bath

RSP 'Goes back to' School, Matfen Hall, Northumberland, 14-16 September 2009


www.ukoln.ac.uk

Presentation outline

• General context: repositories and digital preservation

• Digital preservation overview• Tools:

– Preservation Planning (Plato)– Repository audit (TRAC, DRAMBORA)

• Repositories and the curation of research data– Roles and responsibilities– Infrastructures– Curation challenges


www.ukoln.ac.uk

The repository context (1)

• Repository content is one part of a much wider digital preservation problem– No major specific digital preservation requirements

• Repositories are part of the evolving structure of scholarly communication– Preservation needs to be considered in the same

conceptual space as things like e-journals (e.g., Portico)– There is a commonly-held view that e-prints are just

duplicates of the conventional literature created for immediate access and will not therefore need preservation


www.ukoln.ac.uk


• Repositories can contain many types of materials, but the main focus has usually been on “e-prints” or theses– Broadly text-based (analogues of traditional papers)

• This simplifies preservation requirements– Compared with complex multimedia objects, “the

preservation of e-prints is relatively straightforward from a technical point of view” (Pinfield and James, 2003)

– A large percentage of repository content has (until very recently) been made up of a relatively limited number of formats, e.g.:

• PDF, HTML, MS Word, RTF, TeX, PostScript


www.ukoln.ac.uk


• Repositories are beginning to consider their role in preserving a wider range of content types– Maintain accurate records of the whole research process or

lifecycle (digital curation)– Includes: research data (simulations, materials, the results of

high-throughput instrumentation, open science, etc.), Web pages, Web 2.0 content (blogs, etc.), learning objects, images, time-based media, etc.

– For example: KeepIt project exemplars - research papers, science data, arts, teaching materials and theses: http://preservation.eprints.org/keepit/

• This makes defining preservation requirements for repositories more difficult


www.ukoln.ac.uk


• Repositories need to consider carefully their longer-term objectives and ambitions– Clifford Lynch: “An institutional repository needs to be a

service with continuity behind it … Institutions need to recognise that they are making commitments for the long term”

– This is dependent on institutional support

• Shared infrastructures– Bilateral, regional, national, international– Distributed approaches possible (e.g., SHERPA DP)– A potential key role for national or research libraries (as

with DARE in the Netherlands)


www.ukoln.ac.uk


• Integration of preservation services with repository software– Some experimentation in the PRESERV project:

http://preservation.eprints.org/– Used third-party registries of format information (DROID

and PRONOM from The National Archives) to characterise and validate repository content and to analyse risks

– RSP briefing paper on preservation and storage formats: http://www.rsp.ac.uk/pubs/briefingpapers-docs/technical-preservformats.pdf


www.ukoln.ac.uk

Digital preservation basics

• An ongoing approach to managing digital content based on:– The identification and adoption of appropriate

preservation strategies• Creation or Ingest stages are normally the best time

to ensure that data are fit-for-purpose and “preservable”

– The collection and management of appropriate metadata• Capture of explicit and implicit knowledge, contexts

– The ongoing monitoring of technical contexts and the application of preservation planning techniques

– Continual monitoring of the organisation (audit)


www.ukoln.ac.uk

Technical challenges

• Digital media– Currently magnetic or optical tape and disks, some

devices (e.g., memory sticks)– Uncertain lifetimes

• Hardware and software dependence– Most digital objects are dependent on particular

configurations of hardware and software– Relatively short obsolescence cycles


www.ukoln.ac.uk

Conceptual challenges (1)

• What is an digital object?– Some are analogues of traditional objects, e.g. meeting

minutes, research papers (e-prints)– Others are not, e.g. Web pages, GIS, 3D models of

chemical structures• Complexity• Dynamic nature


www.ukoln.ac.uk


• Three layers:– Physical: the bits stored on a particular medium– Logical: defines how the bits are used by a software

application, based on data types (e.g. ASCII); in order to understand (or preserve) the bits, we need to know how to process this

– Conceptual: things that we deal with in the real world• From: Ken Thibodeau, “Overview of technological

approaches to digital preservation and challenges in coming years.” In: The state of digital preservation: an international perspective (CLIR, 2002): http://www.clir.org/


www.ukoln.ac.uk


• On which of these layers should preservation activities focus?– We need to preserve the ability to reproduce the objects,

not just the bits (would a printout do?)– In fact, we can change the bits and logical representation

and still reproduce an authentic conceptual object (e.g. converting into PDF)

– Increased focus on reuse (e.g, data in tables)

• Authenticity and integrity– How can we trust that an object is what it claims to be?– Digital information can easily be changed by accident or

design


www.ukoln.ac.uk

Some general principles (1)

• Most of the technical problems associated with long-term digital preservation can be solved if a life-cycle management approach is adopted – i.e. a continual programme of active management– Ideally, combines both managerial and technical

processes, e.g., as in the OAIS Reference Model– Many current preservation systems are attempting to

support this approach– Digital preservation strategies need to be seen in this

wider context


www.ukoln.ac.uk


• Preservation needs to be considered at a very early stage in an object's life-cycle

• There is a need to identify 'significant properties'– Recognises that preservation is context dependent, even

user specific (concept of 'designated community')– Helps with choosing an acceptable preservation strategy

• Encapsulation– Surrounding the digital object - at least conceptually - with

all of the information needed to decode and understand it (including software)

– Produces autonomous 'self-describing' objects, reduces external dependencies (linked to the Information Package concept in the OAIS Reference Model)


www.ukoln.ac.uk


• Metadata and documentation is vitally important– Relates to OAIS concepts like Representation

Information and Preservation Description Information– Functions

• Records scientific meaning• Records the research context• Enables the development of finding aids

– Standards are being developed that support digital preservation activities (e.g., the PREMIS Data Dictionary)

• Wherever possible, retain also the original byte-stream


www.ukoln.ac.uk

Digital preservation strategies

• Three main families:– Technology preservation / digital archaeology– Emulation– Migration


www.ukoln.ac.uk

Technology preservation

• The preservation of an information object together with all of the hardware and software needed to interpret it– Successfully preserves the look, feel and behaviour of

the whole system (at least while the hardware and software still functions)

– Severe problems with storage and ongoing maintenance, missing documentation

– May have a role for historically important hardware– May have a shorter-term role for supporting the rescue of

digital objects (digital archaeology)


www.ukoln.ac.uk

Digital archaeology

• Not so much a preservation strategy, but the default situation if there isn't one

• Using various techniques to recover digital content from obsolete or damaged physical objects (media, hardware, etc.)– A time consuming process, needs specialised equipment

and (in most cases) adequate documentation– Considered to be expensive (and risky)– Remains an option for content deemed to be of value

that has not been dealt with in any other way


www.ukoln.ac.uk

Emulation (1)

• Preserving the original bit-streams and application software; running this on emulator programs that mimic the behaviour of obsolete hardware

• Emulators evolve over time– Chaining, rehosting– Emulation Virtual Machines

• Running emulators on simplified 'virtual machines' that can be run on a range of different platforms


www.ukoln.ac.uk

Emulation (2)

• Benefits:– Technique already widely used, e.g. for emulating

different hardware, computer games– Preserves (and uses) the original bits– Reduces the need for regular object transformations (but

emulators and virtual machines may themselves need to be migrated)

– Retains ‘look-and-feel’– May be the only approach possible where objects are

complex or dependent on executable code– Less 'understanding' of formats is needed; little

incremental cost in keeping additional formats


www.ukoln.ac.uk

Emulation (3)

• Challenges:– Do organisations have the technical skills necessary to

implement the strategy?– Preserving 'look and feel' may not be needed for all

objects– It will be difficult to know definitively whether user

experience has been accurately preserved

• Uses:– Promising family of approaches– Needs further practical application and research, e.g.

Dioscuri software (National Library of the Netherlands)


www.ukoln.ac.uk

Migration (1)

• Based on the managed transformation of content:– A set of organised tasks designed to achieve the periodic

transfer of digital information from one hardware and software configuration to another, or from one generation of computer technology to a subsequent one - CPA/RLG report (1996)

– Abandons attempts to keep old technology (or substitutes for it) working

– A 'known' solution used by data archives and software vendors

– Focuses on the perceived content (or significant properties) of objects


www.ukoln.ac.uk

Migration (2)

• Challenges:– Can be labour intensive (batch process, monitoring, QA)– There can be problems with ensuring the ongoing 'integrity

and authenticity' of objects– Transformations need to be documented (typically as part

of the preservation metadata)

• Uses:– Seems to be most suitable for dealing with large collections

of similar objects (e-print repositories?)– Migration can often be combined with some form of

standardisation process, e.g., as part of ingest– A role for repository managers?


www.ukoln.ac.uk

Preservation support on ingest

• Formats can be identified and validated on ingest or deposit into a repository– JHOVE (JSTOR/Harvard Object Validation Environment)– PRONOM, DROID (The National Archives)

• Metadata– Some tools exist for the automatic capture of metadata

• Standardisation on ingest– Perceived wisdom suggests the adoption of open or non-

proprietary standards, e.g. databases structured in XML, uncompressed images, 'preservation friendly' standards like PDF/A


www.ukoln.ac.uk

Choosing a strategy (1)

• Preservation strategies are not in competition– Different strategies will work together, may be value in

diversification– Migration strategies mean difficult choices need to be

made about target formats

• But the strategy chosen has implications for:– The technical infrastructure required (and metadata)– Collection management priorities– Rights management

• Owning the rights to re-engineer software– Costs


www.ukoln.ac.uk

Choosing a strategy (2)

• Plato preservation planning tool (EU Planets project)– A decision support tool that helps users explore the

evaluation of potential preservation solutions against specific requirements and for building a plan for preserving a given set of objects

– Integrates file format identification (using DROID); some migration services; XML-based generic format characterisation using XCL (eXtensible Characterisation Languages)

– http://www.ifs.tuwien.ac.at/dp/plato/intro.html


www.ukoln.ac.uk

Repository audit frameworks (1)

• Repository audit frameworks first developed out of the OAIS Reference Model– OAIS Mandatory Responsibilities (only six of them):

• The main focus was on technical and organisational aspects, e.g.:

– That repositories ensure that preserved information (content) can be understood (independently understandable)

– That documented policies and procedures are being followed

• No clear concept of OAIS compliance (although often claimed by system developers)


www.ukoln.ac.uk


• Trusted Repositories Audit and Certification (TRAC): Criteria and Checklist– RLG-NARA Digital Repository Certification Task Force

checklist, revised (following pilot audits) by the Center for Research Libraries and OCLC

– Criteria cover three main aspects:• Organisational Infrastructure

– Governance and viability, structure and staffing, financial sustainability, contracts, etc.

• Digital Object Management– Ingest, preservation planning, archival storage, etc.

• Technologies, Technical Infrastructure, & Security– Systems and infrastructure, etc.


www.ukoln.ac.uk

TRAC Checklist example page


www.ukoln.ac.uk


• DRAMBORA (Digital Repository Audit Method Based on Risk Assessment)– Digital Curation Centre / Digital Preservation Europe– “Presents a methodology for self-assessment,

encouraging organisations to establish a comprehensive self-awareness of their objectives, activities and assets before identifying, assessing and managing the risks implicit within their organisation“

– Identifying risks and scoring each one on likelihood and impact

– Covers: organisational context, policies, assets, risks, etc.– Online tool (http://www.repositoryaudit.eu/about/)


www.ukoln.ac.uk


• A means of "asking the right questions" about your repository and documenting appropriate procedures and risks

• Both TRAC and DRAMBORA are under consideration by (different) ISO technical committees– External badge of quality (a "certified preservation

repository")

vs.– Management tool for self assessment


www.ukoln.ac.uk

Web links:

• PRESERV project: http://preservation.eprints.org/

• KeepIt project: http://preservation.eprints.org/keepit/

• Plato Preservation Planning tool: http://www.ifs.tuwien.ac.at/dp/plato/intro.html

• DRAMBORA: http://www.repositoryaudit.eu/about/

• RSP briefing paper on preservation and storage formats: http://www.rsp.ac.uk/pubs/briefingpapers-docs/technical-preservformats.pdf


www.ukoln.ac.uk

Questions?

“Pigabyte”

King Bladud’s Pigs in Bath (public art project), Summer 2008

http://www.kingbladudspigs.org/


www.ukoln.ac.uk

Repositories and the curation of research data


www.ukoln.ac.uk

Dealing with research data

• An extremely broad category of material:– “... any information that can be stored in digital form,

including text, numbers, images, video or movies, audio, software, algorithms, equations, animations, models, simulations, etc.” (National Science Board, Long-lived digital data collections, 2005)

– In practice, it can mean almost anything


www.ukoln.ac.uk

Why curate research data? (1)

• Part of the normal research process:– The need for others to validate and replicate research– In some disciplines, supporting data is routinely made

available to reviewers and linked from journal papers– Principles of sharing and openness are firmly embedded

in some disciplines


www.ukoln.ac.uk


• Extrinsic and intrinsic value;– High investment in research– Data can be very expensive to capture and analyse– Data is impossible to recreate once lost– Observational data (by definition) is irreplaceable– Current generations of instruments can gather more data

than can be analysed


www.ukoln.ac.uk


• The potential for creating 'new' knowledge from existing data:– Re-use, re-analysis, data mining– Annotation, e.g. in molecular biology astronomy– Combining datasets in innovative ways, e.g. mapping

biodiversity data onto ecological GIS– “Science 2.0”


www.ukoln.ac.uk


• It is increasingly a requirement of some research funding bodies– Some have quite mature data retention policies (not

necessarily for permanent retention)– Increasing expectation of access to data from publicly-

funded research– OECD Principles and guidelines for access to research

data from public funding (2007)


www.ukoln.ac.uk


• Institutional asset management:– Universities and other research organisations invest very

large sums of money into research activities– Research data is a key output of this activity– It is, therefore, an institutional asset that needs

stewardship


www.ukoln.ac.uk


• Promoting the institution, research group or individual:– Re-use helps promote visibility and 'impact'– Institutions become acknowledged 'centres of

competence'


www.ukoln.ac.uk

Who undertakes preservation?

• Researchers– Indirectly - they have most direct contact with creation

stage, and understand how data can be used– Directly - sometimes responsible for maintaining

community data collections

• Information professionals– Sometimes, but it depends on the context

• IT professionals– Primarily informaticians working with scientists


www.ukoln.ac.uk

Roles and responsibilities (1)

• Long-lived data collections (NSB)– Data authors– Data managers– Data scientists– Data users– Funding agencies

• Dealing with data (JISC)– Scientist– Institution– Data centre– User– Funder– Publisher


www.ukoln.ac.uk


• Scientists– Initial creation and use of data– Expectation of first use and in gaining appropriate credit

and recognition– Responsible for:

• Managing data for life of project• For using standards (where possible)• For complying with data policies• For making the data available in a form that can

(easily?) be used by others


www.ukoln.ac.uk


• Institutions:– Role less clear– Institutional policies may require short-term management

of data• Advocacy and training

– Some institutions are developing repository services• Are rarely currently used for research data• Federated approaches maintain disciplinary

involvement


www.ukoln.ac.uk


• Data centres– Undertakes curation and provides access – Responsible for:

• Selection and ingest• Participating in the development of standards• Protecting the rights of data creators• Supporting ingest and metadata capture• Supporting re-use (tools and services)• Training


www.ukoln.ac.uk


• Users:– Users of third-party data– Responsible for:

• Adhering to any licenses and restrictions on use• Acknowledging data creators and curators• Managing any derived data• Provide feedback to scientists and data centres


www.ukoln.ac.uk


• Funding bodies:– Acting at policy level– Responsible for:

• Considering wider policy perspectives• Developing policies in co-operation with other

stakeholders• Monitoring and enforcing data policies• Support for long-term data management• Support for data curation


www.ukoln.ac.uk

Research data collections (1)

• A typology (1):– From National Science Board report Long-lived digital

data collections (2005)• Research data collections – the products of one or

more focused research projects• Resource or community data collections – collections

that emerge to serve particular subject sub-disciplines

• Reference data collections – serve a broader and more diverse set of user communities


www.ukoln.ac.uk

Research data collections (2)

• Data in “research data collections” is most at risk– A modern version of the “file-drawer problem”– Data stored on personal hard-drives or on media; largely

undocumented– Particular challenge when the data creator has retired or

moved to another institution– Data creators not always aware of its potential value– The reward structure of science is not always helpful


www.ukoln.ac.uk

Curation infrastructures (1)

• Focus on the generic:– Need for a balance between:

• The 'bottom-up' discipline-based drivers that promote the generation of research data

• The policy level, looking to make cost effective investment in curation

– When building Infrastructures, focus on the generic• Storage systems and middleware• Preservation services• Identifying the needs of the wider community


www.ukoln.ac.uk

Curation infrastructures (2)

• The need for collaboration:– Need for 'deep-infrastructure' recognised as far back as

1996 by the Task Force on Archiving of Digital Information

– Digital preservation involves the "grander problem of organizing ourselves over time and as a society ... [to manoeuvre] effectively in a digital landscape" (p. 7)


www.ukoln.ac.uk

Curation challenges: Costs

• NSF Task Force looking at this subject• JISC-funded LIFE (Life Cycle Information for E-

Literature) project is developing a predictive costing tool (http://www.life.ac.uk/)

• JISC-funded study (Keeping research data safe, 2008) focused on research data curation at the institution level– The complex service requirements for curating research

data means that institutions are setting-up federated approaches to repository development

– Currently ingest costs are much higher than long-term storage and preservation costs


www.ukoln.ac.uk

Curation challenges: Scale (1)

• The “digital deluge” in e-Science– New generations of instruments– Computer simulations– Many terabytes generated per day, petabyte scale

computing (and growing)– Cory Doctorow, “Welcome to the petacentre.” Nature,

455, pp 17-21, 4 Sep 2008– Are Institutional Repositories ready for this?

• Digitised content:– Google Book Search (~7 million items)– A role for research libraries?


www.ukoln.ac.uk

Curation challenges: Scale (2)

• Problems of scale are particularly acute in traditional 'big-science' disciplines:– Particle physics (e.g., the Large Hadron Collider)– Astronomy (sky surveys, etc)

• But “smaller experiments will grow the fastest” (Szalay & Gray, Nature, 440, 413-4, 23 Mar 2006)– Bioinformatics, crystallography, engineering design, and

many others

• In some cases it may be cheaper just to generate the data again, e.g. for computer simulations


www.ukoln.ac.uk

Curation challenges: Complexity (2)

• Research data is extremely diverse - not really a single category of material– tabular data, images, GIS, etc.– raw machine output vs, derived data– varying levels of structure (XML, legacy formats, etc.)– many different standards

• Research data is not homogeneous• No one-size-fits-all approach possible


www.ukoln.ac.uk

Curation challenges: Cultures

• Diverse research cultures– Data practices vary widely, even within a single discipline

• Gene sequence data is typically deposited in public databases

• In proteomics, sharing is not so widespread; partly driven by lack of standards, but there is also concern about who have exploitation rights

– Role of commercial interests• Pharmaceuticals, architecture and engineering,

geological prospecting


www.ukoln.ac.uk

The Future ...

• “It is always a mistake for a historian to try and predict the future. Life, unlike science, is simply too full of surprises” - Richard J. Evans, In defence of history (1997, p. 62)


www.ukoln.ac.uk

Further reading

• National Science Board, Long-lived digital data collections: enabling research and education in the 21st century (NSF, 2005) http//www.nsf.gov/pubs/2005/nsb0540/

• Liz Lyon, Dealing with data; roles, rights, responsibilities and relationships (JISC, 2007) http://www.jisc.ac.uk/whatwedo/programmes/digitalrepositories2005/dealingwithdata.aspx

• Neil Beagrie, Jullia Chruszcz, and Brian Lavoie, Keeping research data safe: a cost model and guidance for UK universities (JISC, 2008) http://www.jisc.ac.uk/publications/publications/keepingresearchdatasafe.aspx


www.ukoln.ac.uk

Acknowledgments

• UKOLN is funded by the Joint Information Systems Committee (JISC) of the UK higher and further education funding councils, the Museums, Libraries and Archives Council (MLA), as well as by project funding from the JISC, the European Union, and other sources. UKOLN also receives support from the University of Bath, where it is based.

• More information: http://www.ukoln.ac.uk/


www.ukoln.ac.uk

Thank You!

repositories and digital preservation

Technology

oais reference model

repository audit frameworks

curate research data

research data collections

research papers

research data

repository context

digital preservation