Trends & Challenges in Digital Object Storage

Infrastructure:Notes from the National Digital Stewardship Alliance (NDSA)

Infrastructure Working Group Storage Survey

On behalf of the NDSA:Nancy McGovern, MIT LibrariesJefferson Bailey, Library of Congress

MDOR RoundtableSAA Annual MeetingWednesday, August 8, 2012

Membership:•127 member organizations have joined the NDSA since it was founded in July 2010

•An initiative of the National Digital Information Infrastructure and Preservation Program (NDIIPP) open to any organization committed to the preservation of digital cultural heritage•http://www.digitalpreservation.gov/ndsa

Objective:• A collaborative effort among government agencies,

educational institutions, non-profit organizations, and business entities to preserve a distributed national digital collection for the benefit of citizens now and in the future.

Working GroupsNDSA members commit to participating in one of the five working groups of the Alliance

Infrastructure Working Group: Working to identify and share emerging practices around the development and maintenance of tools and systems for curation and preservation.

Content Infrastructure

Innovation

Standards &

Practices

Outreach

NDSA Infrastructure Storage SurveyGoal:

Obtain a snapshot of current digital object storage practices within NDSA membership

Details: •Conducted between August and November 2011•58 responses from the (then) 74 NDSA members actively preserving digital content

Results:•Published in parts via the NDIIPP blog, The Signal http://blogs.loc.gov/digitalpreservation/•Full report forthcoming

ScopeThe Storage Survey examined activities around 4 topics associated with digital preservation infrastructure:

•Access – How do access requirements impact digital preservation infrastructure?

•Distribution – How are institutions using cloud and distributed storage systems for preservation?

•Fixity – How are institutions ensuring file fixity?

•Administration – How are institutions currently ensuring and strategically planning for digital preservation storage infrastructure.

Access: Modes of Access

Percentage of NDSA Orgs Supporting Each Access Mode

Access: Requirements & SystemsREQUIREMENTS:

53% (31 of 58) reported having a single access requirement (e.g. on-line availability) for all the collections they are preserving.

33% (19 of 58) reported supporting two degrees of access requirements among their collections

14% (8 of 58) reported supporting three or more degrees of access among their collections.

SYSTEMS:

65% (37 of 58) reported providing separate systems for storage and access

35% (20 of 58) reported using the same system for storage and access

Distribution: Cooperative, Contract & Cloud

Storage

Distribution: Geographic & System Preferences

Priorities for storage system functionality:

•More built-in functions like fixity checking

•More automated inventory and retrieval

•More storage space

•Higher performance processing (tasks such as indexing)

File Fixity: Current Practices

File Fixity: Frequency of Checking

Administration: Replication & Media

Administration: Forecasting Storage

Storage Space AmountCurrent Storage For

All CopiesRequirement Anticipated in 3 Years For All Copies

Under 10 TB 17 1010-99 TB 19 15100 to 999 TB 14 161000+ TB (1+ PB) 5 11

Over the next three years my organization plans to… Agree Neutral Disagree

Make significant changes in preservation storage technologies

37 (64%) 10 (17%) 11 (19%)

Will have adequate resources to meet storage requirements

48 (83%) 7 (12%) 3 (5%)

Has a plan to meet our preservation storage requirements

45 (78%) 9 (16%) 4 (7%)

Plans to meet trustworthy digital repository requirements

34 (60%) 20 (35%) 5 (9%)

ChallengesAccess:•Supporting multiple modes of access can require customization of the preservation stack•Access modes largely “conditional” thus requiring tailored workflows

Distribution:•Cloud-storage emerging, but feature-poor•Greater automated features desired in future storage systems

Fixity: •Validation can be processing & I/O intensive•Multiple storage media types can complicate checking•Best practices for frequency of checking are unclear

Administration:•Cost of file and media redundancy•Results showed a bottom-up approach to storage, as opposed to the TRAC top-down approach, though over 60% planned on eventual TRAC compliance•Systems migrations still difficult, expensive

More information about the Alliance is atwww.digitalpreservation.gov/ndsa/

Content Infrastructure

Innovation

Standards &

Practices

Outreach

Read about NDSA work on The Signalblogs.loc.gov/digitalpreservation/

THANKS!Nancy McGovern, MIT LibrariesJefferson Bailey, Library of Congress