Open Repositories 2012, Edinburgh, 11 July 2012
ResourceSync: Web-based
Resource Synchronization
Simeon Warner (Cornell)
Core team -- Todd Carpenter (NISO), Berhard Haslhofer, (Cornell University), Martin Klein (Los Alamos National Laboratory), Nettie Lagace (NISO), Carl Lagoze (Cornell University), Peter Murray (NISO), Michael L. Nelson (Old Dominion University), Robert Sanderson (Los Alamos National Laboratory), Herbert Van de Sompel (Los Alamos National Laboratory), Simeon Warner (Cornell University)
Team members – Richard Jones (JISC/Cottage Labs), Stuart Lewis (JISC/Cottage Labs), Graham Klyne (JISC), Shlomo Sanders (Ex Libris), Kevin Ford (LoC), Ed Summers (LoC), Jeff Young (OCLC), David Rosenthal (Stanford)
Funding – The Sloan Foundation (core team) and the JISC (UK participation)
Thanks for slides from – Stuart Lewis, Herbert Van de Sompel
Synchronize what? • Web resources – things with a URI that can be
dereferenced and are cache-able (no dependency on underlying OS, technologies etc.)
• Small websites/repositories (a few resources) to large repositories/datasets/linked data collections (many millions of resources)
• That change slowly (weeks/months) or quickly (seconds), and where latency needs may vary
• Focus on needs of research communication and cultural heritage organizations, but aim for generality
3
Why?
… because lots of projects and services are doing synchronization but have to roll their own on a case by case basis!
• Project team involved with projects that need this
• Experience with OAI-PMH: widely used in repos but o XML metadata only o Web technology has moved on since 1999
• Data / Metadata / Linked Data – Shared solution?
JISC
Use cases – the basics
JISC
More use cases
Out-of-scope (for now) • Bidirectional synchronization
• Destination-defined selective synchronization (query)
• Special understanding of complex objects
• Bulk URI migration
• Diffs (hooks?)
• Intra-application event notification
• Content tracking
Use case: DBpedia Live duplication • 20M entries updated @ 1/s though sporadic
• Want low latency => need a push technology
Use case: arXiv mirroring • 1M article versions, ~800/day created
or updated at 8pm US eastern time
• Metadata and full-text for each article
• Accuracy important
• Want low barrier for others to use
• Look for more general solution than current homebrew mirroring (running with minor modifications since 1994!) and occasional rsync (filesystem layout specific, auth issues)
Terminology • Resource: an object to be synchronized, a web resource
• Source: system with the original or master resources
• Destination: system to which resources from the source will be copied and kept in synchronization
• Pull: process to get information from source to destination initiated by the destination.
• Push: process to get information from source to destination initiated by the source (and some subscription mechanism)
• Metadata: information about resources such as URI, modification time, checksum, etc. (Not to be confused with resources that may themselves be metadata records)
Three basic needs 1. Baseline synchronization – A destination must be
able to perform an initial load or catch-up with a source
- avoid out-of-band setup; provide discovery
2. Incremental synchronization – A destination must have some way to keep up-to-date with changes at a source
- subject to some latency; minimal: create/update/delete
3. Audit – It should be possible to determine whether a destination is synchronized with a source
- subject to some latency; want efficiency > HTTP HEAD
Baseline synchronization Either
• Get inventory of resources and then copy them one-by-one using HTTP GET
o simplest, inventory is list of resources plus perhaps metadata o inventory format?
or
• Get dump of resources and all necessary metadata o more efficient: reduce number of round trips o dump format?
Audit Could do new Baseline synchronization and compare … but likely very inefficient! Optimize by adding:
• Get inventory and compare with copy at destination o use timestamp, digest or other metadata in inventory to
check content (effort çè accuracy tradeoff) o latency depends on freshness of inventory and time to copy
and check (easier to cope with if modification times included in metadata)
Incremental synchronization Simplest method is Audit and then copy of all new/updated resources, plus removal of deleted resources. Optimize by adding:
• Change Communication – Exchange ChangeSet listing only updates
- How to understand sequence, schedule?
• Resource Transfer – Exchange dumps for ChangeSets or even diffs appropriate to resource type
Change Memory necessary to record sequence or intermediate states.
15
Template to map approaches
JISC
Approaches and technologies
XMPP
AtomPub
SDShare
RSS
Atom
PubSubHubbub
Sitemap
XMPP
rsync
OAI-PMH
WebDAV Col. Syn.
OAI-ORE
DSNotify
RDFsync
Crawl
Push
Pull
SWORD
SPARQLpush
A framework based on Sitemaps • Modular framework allowing selective deployment
• Sitemap is the most basic component of the framework
• Reuse Sitemap form for changesets and notifications (same <url> element describing resource)
• Selective synchronization via tagging
• Discovery of capabilities via <atom:link>!
• Further extension possible
18
19
Baseline Sync with Inventory
Level zero è Publish a Sitemap • Periodic publication of an up-to-date Sitemap is
base level implementation
• Use Sitemap <url> as is with <loc> and <lastmod> as core elements for each Resource
o Introduce optional extra elements to convey fixity information, size, tags for selective synchronization, etc.
• Extend to: o Convey Source capabilities, discovery informatio, locations of
dumps, locations of changesets, change memory, etc. o Provide timestamp and/or additional metadata for the
Sitemap
Two resources, with lastmod times
Two resources, with lastmod times, sizes and digests. The second with a tag also
Sitemap details & issues • Sitemap XML format designed to allow extension
• ResourceSync additions: o Additional core elements in ResourceSync namespace
(digest, size, update information) o Discovery information using <atom:link> elements
• Use existing Sitemap Index scheme for large sets of resources (handles up to 2.5 billion resources before further extension required)
• Provide mapping to RDF semantics but keep XML simple
23
24
Incremental Sync with ChangeSet
ChangeSet
• Reuse Sitemap format but include information only for change
events over a certain period: • One <url> element per change event • The <url> element uses <loc> and <lastmod> as is and
is extended with: • an event type to express create/update/delete • an optional event id to provide a unique identifier for the
event. • can further extend to include fixity, tag info, Memento
TimeGate link, special-purpose access-point, etc. • Introduce minimal <urlset>-level extensions to support:
• Navigation between ChangeSets via <atom:link> • Timestamping the ChangeSet
25
Expt: arXiv – Inventory and ChangeSet • Baseline synchronization and Audit (Inventory):
o 2.3M resources (300GB content) o 46 sitemaps and 1 sitemapindex (50k resources/sitemap) o sitemaps ~9.3MB each -> 430MB total uncompressed;1.7MB
each -> 78MB total if gzipped (<0.03% content size)
• Incremental synchronization (ChangeSet): o arXiv has updates daily @ 8pm so create daily ChangeSet o ~1k additions and 700 updates per day o 1 sitemap ~300kB or 20kB gzipped, can be generated and
served statically o keep chain of ChangeSets, link with <atom:link>
27
Incremental Sync with Push via XMPP
Change Communication: Push via XMPP • Rapid notification of change events via XMPP
PubSub node; one notification per event
• Each change event is conveyed using a Sitemap <url> element contained in a dedicated XMPP <item> wrapper
• Use same resource metadata (e.g. <loc>, <lastmod>) and same extensions as with changesets
• Multiple change events can be grouped into a single XMPP message (using <items>)
Expt: LiveDBpedia with XMPP Push • LANL Research Library ran a significant scale
experiment in synchronization of the LiveDBpedia database from Los Alamos to two remote sites using XMPP to push change notifications
o Push for change communication only, content then obtained with HTTP GET
• Destination sites were able to keep in close synchronization with sources
o Maximum queued updates <400 over 6 runs with 100k updates; and bursty updates averaging ~1/s
o Small number of errors suggests use for audit in many real-life situations
Dumps Optimization over making repeated HTTP GET requests for multiple resources. Use for baseline and changeset. Options:
1. ZIP+Sitemap o simple and ZIP very widely used o consistent inventory/change/set format o con: “custom”
2. WARC o designed for exactly this purpose o con: little used outside web archiving community
31
Sitemaps + XMPP + Dumps
Timeline and input • July 2012 – First draft of sitemap-based spec (SOON)
• August 2012 – Publicize and solicit feedback (will be NISO email list)
• September 2012 – Revise, more experiments, more feedback
• December 2012 – Finalize specification (?)
• NISO webspace
• Code on github: http://github.org/resync/simulator