21 st century archives: architecting the future

21st Century Archives:Architecting the Future

Adam Jansen, CRM, MIT, MCP, CDIADKives Consulting

adam@dkives.com

NDPP -- Mar 2009

BACKGROUND

Washington State Digital Archives• 80+ million public records in the system• 950 Researchers per day• 130+ foreign countries in one month• 25 categories of records (land, court, photos,

web pages, email, audio recordings)• 550 state web pages spidered– 2.5 Million ‘web files’

• 99.99+% Uptime• 3300 state and local agencies as potential

customers

Standards Driven

• Open Archival Information System – ISO Standard for electronic records archiving

• DOD 5015.2 – ISO Standard for Records Management Applications

• InterPARES – International effort to define requirements for e-archiving

• INCITS/V1 committee for OpenXML

DA Responsibilities

• Negotiate and accept records from producers• Control to level needed for long term• Understand target audience• Ensure independence, understandable• FOLLOW DOCUMENTED PROCESS• Make records accessible

What is ‘Archiving’ in the Electronic Age?

Protecting machine readable records of enduring legal, historical or fiscal value from loss, alteration, deterioration and technological obsolescence in a environment independent from that which produced the record.

Shifting Media• Historically records were stored on paper, kept in filing

cabinets– When the cabinet was full, records sent to file room

• Now records stored electronically on computers– When the computer is ‘full’ – add more hard drives

Basic skills to manage and maintain records have been lost, replaced by infinite storage

So the question becomes… who takes care of the records, and do they have the knowledge?

IMPLEMENTING A DIGITAL ARCHIVES

Brick and Mortar

• Delivery• Loading Dock• Processing Area• ‘The Stacks’• Research Room

DELIVERY TRUCK

Records Transportation

• Preferred method is Internet delivery– Can be VERY slow

• Also accept on optical, hard drive, tape• Have multiple layers of politics to satisfy• Large number of small files creates problems• Provide removable media devices to

partner/clients

10 TB transfer example• FIREWIRE800 @ 100MBS = 27.8 Hours• FIREWIRE400 @ 50MBS = 55.6 Hours• USB 1.1 @ 1.5 MBS = 1851.9 Hours• USB 2.0 @ 60 MBS = 46.3 Hours• SATA 1 @ 150 MBS = 18.5 Hours• SATA 2 @ 300 MBS = 9.3 Hours• LTO2 @ 30 MBS = 92.6 Hours• SDLT 320 @ 16 MBS = 173.6 Hours• Magneto Optical @10 MBS = 277.8 Hours

PDI Example

Ethernet adapter Local Area Connection: Connection-specific DNS Suffix . : annex.co.franklin.wa.us Description . . . . . . . . . . . : Intel(R) PRO/100 VE Physical Address. . . . . . . . . : 00-0D-60-3C-22-34 DHCP Enabled. . . . : Yes Autoconfiguration Enabled . . . . : Yes IP Address. . . . . . . . . . . . : 172.30.7.39 Subnet Mask . . . . . . . . . . . : 255.255.255.0Default Gateway . . . . . . . . . : 172.30.7.1 DHCP Server . . . . . . . . . . . : 172.30.7.2 DNS Servers . . . . . . . . . . . : 172.30.7.2, 198.239.73.3 Primary WINS Server . . . . . . . : 172.30.7.2 Secondary WINS Server . . . . . . : 198.239.73.3 Lease Obtained. . . . . . . . . . : Tuesday, August 23, 2005 8:34:25 AM Lease Expires . . . . . . . . . . : Thursday, September 22, 2005 8:34:25 AM

LOADING DOCK

Landing Pad

• Data must be clean• Quarantine and recheck• Outside of permanent repository• Ease to send, not remove• Provide secure storage until processing

PROCESSING AREA

Clean and Prep Records

• Verify records for completeness• Send rejects back to agency• Assemble records for ingestion• Create/note missing metadata• Workflow for file conversion– Preservation file format– Presentation file format

• Move into database, backup

THE STACKS

Archival Storage• Receive Data– Move to ‘permanent’ storage

• Expect frequency

– Confirmation and location

• Manage Storage– Based on policy, statistics– Monitor error logs– Ease of expanse, growth on demand

• Replace Media– Sampling– Life expectancy, technology changes

Management of Storage

• Store records in open, non-proprietary method

• The only constant is change• Maintain authenticity, integrity• Allow for future migration, conversion• MUST be easy to expand and change• Flexibility is the key!!!

RESEARCH ROOM

17th Century“So long as men can breath, and eyes can see,So long lives this, and this gives life to thee.”

- William Shakespeare, Sonnet XVIII (1609)

21st Century As long as the media is still readable, a drive available,

the software can be loaded, the hardware still runs and file can be found!

Questions?

Adam Jansenadam@dkives.com