the preserve-or-forget reference model and framework (wp8 forgetit 1st year review)

Concise Preservation by combining Managed Forgetting and Contextualized Remembering

Francesco GalloEURIX

WP8 PresentationThe Preserve-or-Forget Reference Model and Framework

ForgetIT 1st Review Meeting, April 29-30, 2014 Kaiserslautern, Germany

WP Objectives (from DoW) ● integrate project components into a technologically coherent framework● adopt flexible and extensible solutions● define a PoF reference model supporting ForgetIT concepts

Focus of Year 1• design of the PoF framework architecture• technology assessment• identification of project components and early integration• started working on definition of PoF model

ForgetIT Project GA600826, 1st Review Meeting, Kaiserslautern, April 2014

Objectives of WP and Year 1 Focus

Design of the architecture for the Preserve-or-Forget Framework

Assessment of technologies for PoF middleware and AIS

Definition of components from all WPs, preliminary integration

Analysis of requirements for PoF reference model

Testbed setup and integration plan

PoF framework prototype, synergetic preservation workflow

• ForgetIT Project GA600826, 1st Review Meeting, Kaiserslautern, April 2014

Achievements in Year 1


Preserve-or-Forget Architecture (D8.1)


PoF Framework Components and APIs (D8.1)

middlewareapplications archive

cloud storage

WP3,WP4,WP5,WP6WP10

WP9 WP8

WP7


PoF Middleware Components

Shared components (general tasks)● ID Manager, Metadata Repository, Scheduler, Context-Aware

Preservation Manager

Other components (core ForgetIT principles) ● Forgettor, Extractor, Condensator, Contextualizer, Navigator,

Collector, Archiver

For each component full description provided in D8.1

Many components available as prototypes, already integrated

Evaluated several candidates for implementing the Archive ● Archivematica, FedoraCommons, DSpace, RODA, P4, iRODS, …

Assessment criteria: ● open-source license, support for ForgetIT data types, TRL,

integration with PDS, language and technologies, documentation,

supporting community, …

DSpace selected as the candidate implementation of PoF Archive● widely adopted and actively maintained, supports all ForgetIT data

types, integrates with cloud storage solutions and other platforms● extensible with custom add-ons, periodic digital curation tasks,

validation upon ingest, user profiles, ...


Assessment of OAIS platforms (D8.1)


Archive: DSpace admin interface


Archive: DSpace AIP preview

Enterprise Service Bus● communication layer for all integrated components

Enterprise Integration Patterns● distributed applications and services developed by all WPs● leverage best practices in enterprise application integration

Message Oriented Middleware● message-based communication layer: asynchronism, routing,

transformation, decoupling, reduced integration complexity

Candidate MOM implementation: Apache ServiceMix


Technologies for PoF Middleware (D8.3)


PoF Middleware: Message Oriented Middleware

© F. Munz, Middleware and Cloud Computing, 2011

© D.A. Chappel, Enterprise Service Bus, 2004

PoF Middleware includes rule-based routing and mediation engine

implementing all Enterprise Integration Patterns (EIPs)

Seamless integration with messaging system

Pattern Examples: Reply/Forward, MessageRouter


Workflows: rule-based message routing


PoF Middleware: messaging system


PoF Middleware: preserved items

Synergetic Preservation and Managed Forgetting Workflow

Resource restore from Applications (after processing in the storage)

Periodic Curation Tasks (fixity and format checks for bitstreams,

metadata checks for link consistency and completeness, …)

Format migration: leverage PDS computational storage (Storlets)

Metadata migration: DSpace provides tools to convert metadata from

one schema to another, intermediate mapping, extensible

Storage management: integration of DSpace Archive and PDS


Preservation in PoF Framework


Basic Synergetic Preservation Workflow (D8.1)

applications

middleware

archive

cloud storage

Archive: DSpace data model for SIP and AIP (items, collections,

communities), OAIS functional entities implemented (Ingest/Access,

Administration, Data Management, Preservation Planning and

Archival Storage shared with PDS)

PoF Middleware: SIP creation and ingest, DIP access, smooth bi-

directional transition from Applications to AIS, workflow management

PDS: Archival Storage, preservation actions close to data, internal

data model (tenant, docket, aggregation)


OAIS and the PoF Framework

Expected outcome at the end of the project: extend OAIS model to

support ForgetIT approach to preservation

OAIS specification provides: functional model, information model and

model for information package transformation


PoF Reference Model


PoF Reference Model -2

Work in progress: identify model requirements based on ForgetIT

principles and scenarios, analyze available internal models (e.g.

contextualization model) and compare with OAIS

Evaluate emerging digital preservation standards (e.g. MPEG MP-AF)

Challenging activity throughout the whole project lifetime, and an opportunity

Subversion repository and Trac for issue reporting system

ForgetIT Private Network (VPN)

KVM for virtualization (components and services deployed as VMs)

Shared data storage for test samples


Testbed environment and collaborative development

Collaboration with Presto4U Coordination Action (FP7)● assessment of digital preservation platforms and tools adopted by

different communities of practice (archivists, museums, broadcasters, ...)

Lead of MPEG MP-AF Working Group● Multimedia Preservation Application Format● evaluation of standard metadata formats for digital preservation● new standard ISO/IEC 23000-15 for interoperable digital

preservation format


Dissemination activities

Thank you for your attention!

the preserve-or-forget reference model and framework (wp8 forgetit 1st year review)

Technology

review meeting

pof framework components

pof middleware components

components available

forgetit concepts

forgetit data types

early integration

ais definition of components