How much preservation do I get if I do absolutely nothing?

Using the Web Infrastructure for Digital Preservation

Michael L. Nelson, Frank McCown, Joan A. Smith, Martin KleinOld Dominion University

Norfolk VA, USA

{mln,fmccown,jsmit,mklein}@cs.odu.edu

Media Production Berlin 2006

Berlin, Germany

December 8, 2006

Research supported in part by NSF, Library of Congress and Andrew Mellon Foundation

Preservation: Fortress Model

1. Get a lot of $

2. Buy a lot of disks, machines, tapes, etc.

3. Hire an army of staff

4. Load a small amount of data

5. “Look on my archive ye Mighty, and despair!”

image from: http://www.itunisie.com/tourisme/excursion/tabarka/images/fort.jpg

Five Easy Steps for Preservation:

Alternate Models of Preservation

• Lazy Preservation– Let Google, IA et al. preserve your website

• Just-In-Time Preservation– Wait for it to disappear first, then a “good enough” version

• Shared Infrastructure Preservation– Push your content to sites that might preserve it

• Web Server Enhanced Preservation– Use Apache modules to create archival-ready resources

image from: http://www.proex.ufes.br/arsm/knots_interlaced.htm

Lazy Preservation

Research Questions

• How much digital preservation of websites is afforded by lazy preservation?– Can we reconstruct entire websites from the WI?– What factors contribute to the success of website

reconstruction?– Can we predict how much of a lost website can be

recovered?– How can the WI be utilized to provide preservation of

server-side components?

Warrick: Crawling the Crawlers

• Is website reconstruction from WI feasible?– Web repository: G,M,Y,IA– Reconstructed 24 websites

• How long do search engines keep cached content after it is removed?

SE Caching Experiment

• Create html, pdf, and images• Place files on 4 web servers• Remove files on regular schedule• Examine web server logs to determine

when each page is crawled and by whom• Query each search engine daily using

unique identifier to see if they have cached the page or image

Joan A. Smith, Frank McCown, and Michael L. Nelson. Observed Web Robot Behavior on Decaying Web Subsites. D-Lib Magazine, February 2006, 12(2)

Caching of HTML Resources - mln

Reconstructing a Website

Warrick

Starting URL

Web Repo

Original URL

Results page

Cached URL

Cached resourceFile system

Retrieved resource

1. Pull resources from all web repositories

2. Strip off extra header and footer html

3. Store most recently cached version or canonical version

4. Parse html for links to other resources

How Much Did We Reconstruct?

A

“Lost” web site Reconstructed web site

B C

D E F

A

B’ C’

G E

F

Missing link to D; points to old resource G

F can’t be found

Reconstruction Diagram

added 20%

identical 50%

changed 33%

missing 17%

Websites to Reconstruct

• Reconstruct 24 sites in 3 categories:1. small (1-150 resources) 2. medium (150-499 resources)3. large (500+ resources)

• Use Wget to download current website• Use Warrick to reconstruct• Calculate reconstruction vector

Results

Frank McCown, Joan A. Smith, Michael L. Nelson, and Johan Bollen. Reconstructing Websites for the Lazy Webmaster, Technical Report, arXiv cs.IR/0512069, 2005.

Web Repository Contributions

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Reconstructed websites

Contribution

Yahoo

IA

MSN

Google

Warrick Milestones

• www2006.org – first lost website reconstructed (Nov 2005)

• DCkickball.org – first website someone else reconstructed without our help (late Jan 2006)

• www.iclnet.org – first website we reconstructed for someone else (mid Mar 2006)

• Internet Archive officially “blesses” Warrick (mid Mar 2006)1

1http://frankmccown.blogspot.com/2006/03/warrick-is-gaining-traction.html

Shared Infrastructure Preservation

(slightly less lazy)

Shared, Existing Infrastructure

• Can we (re)use existing installed network infrastructure for preservation purposes?

Who has the Bigger Fortress?

Research Objective• Premise: use common Internet Protocol implementations to

replicate repository contents• Inject the contents of an OAI-PMH repository directly into:

– Email (SMTP)– Usenet News (NNTP)

• Instrument existing email, news servers• Use mod_oai (www.modoai.org) to do resource harvesting

– complex object formats (e.g. MPEG-21 DIDL) used to encode the resources as “lumps of XML”

– results are generalizable to any repository system

• Analyze testbed, simulate very large collections

complex objects

Prototype Architecture

Test Repository

• Website with 72 files – HTML, PDF, PNG, JPEG, GIF– 1KB - 1.5 MB

• Used a script to harvest the MPEG-21 DIDLs, and then:– attach to outbound email mesgs– post to a moderated newsgroup

(repository.odu.test1)

Email Headers

OAI-PMH & HTTPheaders

base64 encoded DIDL

original email mesg

News Posting

OAI-PMH & HTTPheaders

base64 encoded DIDL

Simulation Parameters

• Repository– 100,000 items– 1MB/item– 100 daily additions– 400 daily updates

• Time– 2000 days (5.5 years)

• Email– granularity=1

– follows ODU power law example

• News– servers hold contents

for 30 days

News Policies

NNTP Results

SMTP Policies

• passive, “piggybacking”• History list of receiver domains

– not maintained; history pointer off» duplicates

– maintained; history pointer on» no duplicates

• Granularity Filter for emails– every Gth email will be processed

SMTP Results no history pointer with history pointer

G = 1

Summary

• Shared Infrastructure Preservation provides a communications channel with unknown, future trading partners– SMTP approach is only feasible for “advertising” the existence of

the repository– NNTP approach is promising for holding content

• Lazy Preservation has been used to restore several dozen websites– but is it an archival strategy? depends on your tolerance for risk– prediction: search engines will see preservation as a business

opportunity