ibm haifa research lab © 2008 ibm corporation contacts: simona cohen, michael factor, dalit naor

IBM Haifa Research Lab

© 2008 IBM Corporation

Contacts: Simona Cohen, Michael Factor, Dalit Naor

http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

Preservation DataStores (PDS): A demonstration of Preservation Aware Storage


© 2008 IBM Corporation2 http://www.haifa.il.ibm.com/projects/storage/ltdp/index.shtml

PDS Overview

OAIS–based preservation-aware storage, media-agnostic, and generic storage to support logical preservation

Manage preservation specific metadata– Compute fixity– Update technical provenance– Manage PDI RepInfo– Ensure referential integrity

Storlet container– Module container that can execute restricted modules with predefined interfaces for

data intensive functions, e.g., transformations, fixity calculations– Optimal scheduling– Update PDS modules, e.g., fixity algorithm, packaging format

Managing availability/ data loss– Physically co-locate data and metadata– Cluster related AIPs on the same media unit, based upon their relative importance

AIP identifier generation – globally unique identifiers

Content DataObject

Representation Information

Reference

Context Fixity

Provenance

Content Information Preservation Descriptive Information



PDS Architecture

Layered approach is based on open standards: OAIS, XAM, OSD

In CASPAR, all layers are utilized. In other embodiments, only some layers can be used

Utilizes XAM to provide logical abstraction of containers (XSets)

Offloads preservation functionality to storage

ingestAIP, accessAIP, transformAIP, getPreservationPolicy



AIP Representation in the Preservation Engine

The basic building-block of an AIP is the Information Object

– A single instance of data is a byte stream

– Zero or more instances of RepInfo to interpret the data

RepInfo is also an AIP– Shared resource with

unique ID

– RepInfo recursive network

– Keep the design simple



Preservation of an AIP

Ingest AIP– Storing an AIP in PDS

Bit and logical preservation of an AIP– Bit migrations

– Format migrations - data transformation• Transformation modules are packed as AIPs and preserved• Transformation result is a new version for the original AIP

– Migrations are documented as Provenance records

– During migrations PDS performs operations on AIP• Update PDI (e.g., fixity calculation, additional provenance events)• Execute previously loaded storlets

Access AIP– Retrieval of an AIP

• By retrieval time, the original AIP may have several versions and copies



Demo



MST DataNatural Environment Research Council (NERC) Mesosphere-Stratosphere-Troposphere (MST) Radar at Aberystwyth

Provides measurements of vertical and horizontal components of the wind

Highly complex data

Preserved for the atmospheric scientific community

Needs long-term preservation for research purposes

Community is highly dependent on interoperability



British Atmospheric Data Centre and Self-Describing Data Formats

BADC strongly advocate the use of self-describing files to capture provenance and semantic information

Two self-describing file formats compete in the Atmospheric Science Community – NASA AMES (ASCII)

– NetCDF (binary)

The current user community finds NetCDF more suitable



10 years later

20 years later

Ingest AIP

Load and apply transformation

Access and view transformed data

View AIP

Demo Flow



pds_view

Video:



Ingest AIP flow in PDS

Preservation Engine

PDS web service

XAM

XAM to OSD (VIM)

OSD

ingest (AIP)

XUID

AIP ID

create XSet

write fields (AIP data)

commit

create OSD objects

write (XSet data)

Unpack AIP Generate AIP ID Compute fixity Add “ingest”

provenance event

store (OID, XUID) mapping persistently

store (XUID, AIP ID) mapping persistently



10 years later

20 years later

Ingest AIP



View AIP

Demo Flow



pds_ingest

Video:



Format Change

Converting NetCDF to NASA-AMES

– The MST NetCDF file version is no longer supported by the NetCDF foundations libraries, and files are not backwards compatible with this NetCDF version

– The skills base of the atmospheric scientist in the future is heavily biased towards the use of ASCII-based file formats

– The NASA-AMES format has become the preferred data format of the atmospheric scientists’ community

Converting NetCDF to PNG

– Provide additional quick image view of the data for users with limited access permissions



Load transformation flow in PDS

Preservation Engine

PDS web service

loadTransformation (AIP)

AIP ID

Register transformation Ingest transformation AIP



Transform AIP flow in PDS

PDS web service

transformAip (targetAIPID, transformationAIPID)

AIP ID

Access transformation AIP Access target AIP Execute transformation Ingest result as new AIP, a

version of the target AIP

Preservation Engine



10 years later

20 years later

Ingest AIP



View AIP

Demo Flow



pds_xform

Video:



Access AIP flow in PDS

Preservation Engine

PDS web service

XAM

XAM to OSD (VIM)

OSD

access (AIP ID)

read OSD objects

AIP data

AIP

XSet data

open XSet (XUID)

read fields

Lookup XUID by AIP ID

Lookup OSD OID by XUID

Validate AIP ID Validate fixity Add “access”

provenance event Package AIP



Demo Flow

10 years later

20 years later

Ingest AIP



View AIP



pds_access

Video:



Thank you!

Haifa preservation team:

Simona Cohen, Michael Factor, Aner Hamama, Ealan Henis, Dalit Naor, Petra Reshef, Shahar Ronen

ibm haifa research lab © 2008 ibm corporation contacts: simona cohen, michael factor, dalit naor

Documents

ibm haifa research lab

ibm corporation demo

ibm corporation contacts

aip xuid aip id

aip flow

aip representation

aip load

fields aip data commit