The New DRS: Plan for Metadata Migration

Harvard Library & Library Technology ServicesFebruary 26, 2014

Agenda

Welcome and introduction …... Franziska Frey

Migration challenges .…………... Randy Stern

Creating the plan ………………..... Kate Bowers

Walkthrough of plan …………….. Andrea Goethals

Imaging Services:

minimizing disruption……….… Bill Comstock

Wrap-up & next steps ………...… Kate Bowers, Andrea Goethals

Q & A ……………………………….…... All

WELCOME & INTRODUCTIONFranziska Frey, Tracey Robinson

The DRS Advisory Group…

…provides oversight and guidance during the rollout phase of the DRS2 project and ensures that the user community of active DRS depositors and content owners contributes to decisions about the rollout.

Amy BensonKate BowersBill ComstockFranziska Frey (chair)

Andrea GoethalsWendy GogelTracey Robinson Randy Stern

Why a New DRS?

• Upgrade to best-in-breed technologies• Adopt digital preservation best practices and

standards• Preserve metadata better• Improve collection management• Support preservation planning & activities• Improve access to content & metadata• Support more formats & genres

Preservation Capability Before and After the DRS2 Project

Level One Level Two Level Three Level Four

Storage & Geographic Location

File Fixity and Data Integrity

Information Security

Metadata

File Formats

= already compliant = will be compliant after the DRS2 project

Based on the NDSA Levels of Digital Preservation

Evolution of the DRS

2000 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 20122001

Current DRS in

production

New DRS in productionDRS enhancements

New DRS infrastructuredevelopment

2013 2014 2015

New DRS metadata migration

& user adoption

New DRS - Completed

2009 2010 2011 2012

convened DRS

Advisory Group

software in production

2013 2014 2015

users trained,phase 1

hardware in production

migrated content to new hardware

InfrastructureDevelopment

Metadata Migration

& User Adoption

Fedora assessment

DuraCloud pilot test

early release beta 1 beta 2

beta 3

first object deposited to the new

DRS

New DRS - Upcoming

2009 2010 2011 2012 2013 2014 2015

InfrastructureDevelopment

Metadata Migration

& User Adoption

metadata migration tools

created

metadata migrated

users moved

MIGRATION CHALLENGESRandy Stern

Why “Metadata” Migration?

Why not“content” migration?

Pre-migration

DRS Content

Current DRS

Database

Post-migration

DRS Content

Current DRS

Database New DRS Database

New DRS Index

New DRS Object Descriptors

New DRS Data Model

• Not a simple metadata conversion• A new DRS object is a logical intellectual

entity that unifies multiple DRS files– Still image objects - archival and production

masters, and deliverables including thumbnails – Audio objects - archival and production

masters and deliverables– PDS objects - page image and text files

Object Descriptors

• METS files generated for each object– Standards-based internal schemas (PREMIS,

MODS, MIX, etc.)

• Metadata gathered from multiple sources– Current DRS database– Every content file– HOLLIS records

Technical Challenges

• Many formats– Images, audio, text, digitized books, web sites,

documents, biomedical image stacks, opaque files

• Unique migration rules per format – technical metadata, roles, relationships

• Large (>5000 file) PDS documents• 45+ million DRS files

Technical Challenges

• At 1 sec/file, 45 million files would take 520 days!

• We are designing the migration software tools for parallel processing

• We are configuring multiple servers to run the migration

CREATING THE PLANKate Bowers

Formulating a Migration Strategy

• Analysis of:– DRS content

• Technical (relationships, etc. for building objects)• Volume and type by repository

– metadata for mapping– user activity in the DRS– survey of highest volume, active users– training and testing registration lists

Migration Strategy Factors

• Combines needs of users with technical requirements

• User sequencing will be based on:– Current deposit & administrative activity – Level of preparation (training and participation

in beta testing)

WALKTHROUGH OF PLANAndrea Goethals

Migrating Content in 5 Stages

Migrate 1st : Tier 1 contentMigrate 2nd: Tier 2 contentMigrate 3rd: Tier 3 contentMigrate 4th: Tier 4 contentMigrate 5th: Tier 5 content

Migrating Content in 5 Stages

Migrate 1st : Tier 1 contentMigrate 2nd: Tier 2 contentMigrate 3rd: Tier 3 contentMigrate 4th: Tier 4 contentMigrate 5th: Tier 5 content

simpler objects

more complex objects

Migrating Content in 5 Stages

Migrate 1st : Tier 1 contentMigrate 2nd: Tier 2 contentMigrate 3rd: Tier 3 contentMigrate 4th: Tier 4 contentMigrate 5th: Tier 5 content

dependenciesbetween tiers

dependencieswithin tiers

Migrating Content in 5 StagesTier Content

1 Text (Methodology, ESRI World File), Document, Color Profile, Target Image

2 PDS Document, Still Image

3 Audio, Text (SMIL)

4 Web Harvest, Opaque Container

5 Biomedical Image; Google Document Container 1, 2, 3

Migrating Content in 5 StagesTier Content

1 Text (Methodology, ESRI World File), Document, Color Profile, Target Image

2 PDS Document, Still Image

3 Audio, Text (SMIL)

4 Web Harvest, Opaque Container

5 Biomedical Image; Google Document Container 1, 2, 3

Migrating Content in 5 StagesTier Content

1 Text (Methodology, ESRI World File), Document, Color Profile, Target Image

2 PDS Document, Still Image

3 Audio, Text (SMIL)

4 Web Harvest, Opaque Container

5 Biomedical Image; Google Document Container 1, 2, 3

Tiers 1, 3, 4, 5: Migrate across all DRS owner codes at one timeTier 2: Migrate one DRS owner code at a time

Tier 2: Sequence by DRS Owner Code

• Migrate just your unit’s PDS document and still image content

• Minimize the amount of time the content you manage the most is in 2 different systems

DRS Owner Codes to be Migrated

DIV.LIBR

FHCL.HOUGH

FHCL.MUSI

GSD.LIBR

RAD.ARCH

RAD.SCHL

FHCL.JUD

FHCL.FAL

FHCL.MAPS

FMUS.MCZ

HLS.LIBR

HUL.ARCH

HUAM.MUSE

HBS.BAKR

VIT.BERE

HUL.PRES

FMUS.GRAY

HPPM.PIRC DOAK.RESLIB

DOAK.MUS

FCOR.REISCH

FMUS.ARN

HLNC.LIBRARB.AAHOD

DOAK.LIBR

FCOR.FORST

FCOR.WOLBACH

FMUS.FARL

FMUS.HUH

FMUS.ORC

FMUS.PEAB

HMS.COUNTHPRE.WARD

HUAM.SARDISHUL.GGL

HUL.OIS

FCOR.CARP

FCOR.HCO FCOR.URI

FHCL.CAB

FHCL.COLL

FHCL.DAVIS

FHCL.ENV

FHCL.FUNG

FHCL.GOV

FHCL.LITTFHCL.MED

FHCL.SLV

FHCL.TOZ FHCL.YENCH

FMUS.SEM

FMUS.WARE

GSE.GUTMN

KSG.LIBR

Timing

• Current estimates: – Building & testing migration tools: Now– Begin Tier 1 content: Spring 2014– Begin Tier 2 content: Summer 2014

• Units will be contacted about their Tier 2 migration schedule

After Your Tier 2 Migration

• You, and anyone depositing on your behalf, will begin depositing only to the new DRS

• All of your management tasks will be done only in the new DRS

IMAGING SERVICES: MINIMIZING DISRUPTION

Bill Comstock

Minimizing Disruption

• Testing by Imaging Services• Uninterrupted services• Migration sequencing• Participating as a “pioneer”

Testing by Imaging Services

Alpha and beta testing:• Depositing processes• DRS content maintenance tools

– Searching and assembling content for download

– Editing PDS objects

Uninterrupted Services

Providing services before and after your migration • Content needs to be deposited• Content needs to be searched • Content needs to be assembled

• may need to be edited• may need to be downloaded

Migration Sequencing

• We will synchronize deposits with your migration– start depositing for you in the new DRS

after your Tier 2 content is migrated

Imaging Services as Pioneers

As pioneers, we:• Learn to use the new tools• Refine the new depositing workflows• Identify bugs• Suggest improvements• Create a group of local experts that can support those

that follow

We’ll wear the scars so that you can stay pretty!

WRAP-UP AND NEXT STEPSKate Bowers and Andrea Goethals

Nine Pioneers

• Limited number of first depositors to new DRS• Factors

– New DRS-ready content from new systems • EAS (Electronic Archiving Service), ACORN (Weissman

Preservation Center conservation treatments), DASH (for ETD)

– Prepared and trained staff– No content to migrate

• HUA example: opaque objects

First Deposit in the New DRS

大藏經 Da Zang Jing - Buddhist sutra, Qing dynasty (1644 -1911), China, Tibetan language

Email List

drs-migration@hulmail.harvard.edu

http://hul.harvard.edu/ois/systems/drs/drs2.html

Coming Attractions

• Open meetings– Technical aspects brown bag (March)– Digital preservation & DRS intro (Summer)

• Training and instruction– Refresher training– New training– Onsite assistance

Q & AThanks!