Building Scalable Web Archives

Florent Carpentier, Leïla Medjkoune Internet Memory Foundation

IIPC GA, Paris, May 2014

Internet Memory Foundation

Internet Memory Foundation (European Archive)Established in 2004 in Amsterdam and then in Paris:•Mission: Preserve Web content by building a shared WA platform•Actions: Dissemination, R&D and partnerships with research groups and cultural institutions•Open Access Collections: UK National Archives & Parliament, PRONI, CERN The National Library of Ireland, etc.

Internet Memory ResearchSpin-off of IMF established in June 2011 in Paris•Mission: Operate large scale or selective crawls & develop new technologies (processing and extraction)

Internet Memory FoundationFocused crawling:•Automated crawls through the Archivethe.Net shared platform•Quality focused crawls :• Video capture (You Tube channels), Twitter crawls,

complex crawls

Large scale crawling•Inhouse developed distributed software •Scalable crawler: MemoryBot•Also designed for focused crawl and complex scoping

Research projects

Web Archiving and PreservationLiving Web Archives (2007-2010)Archives to Community MEMories: (2010-2013)SCAlable Preservation Environment (2010-2013)

Webscale data Archiving and Extraction✓Living Knowledge (2009-2012)✓Longitudinal Analytics of Web Archive data (2010-2013)

MemoryBot design (1)• Started in 2010 with the support of the LAWA

(Longitudinal Analytics of Web Archive data) project

• URL store designed for large-scale crawls (DRUM)• Built in Erlang: distributed and fault-tolerant

system language• Distributed (consistent hashing)• Robust: topology change adaptation, memory

usage regulation, process isolation

MemoryBot design (2)

MemoryBot performance

• Good throughput and slow decrease• 85 resources written per second, slowing to 55 after 4

weeks on a nine 8-core servers cluster (32 GiB of RAM)

MemoryBot counters

MemoryBot counters

MemoryBot – quality

• Support of HTTPS, retries on server failure, configurable URL canonicalisation

• Scope: domain suffixes, language, hops sequence, white lists, black lists

• Priorities• Trap detection (URL pattern identification, within

PLD duplicate detection)

MemoryBot – multi-crawl

• Easier management• Politeness observed across different crawls• Better resource utilisation

IM InfrastructureGreen datacenters•Through a collaboration with NoRack•Designed for massive storage (petabytes of data)•Highly scalable/low consumption •Reduces storage and processing costs

Repository :•HDFS (Hadoop File System): Distributed, fault-tolerant file system•Hbase. A distributed key-value index (temporal archives)•MapReduce: A distributed execution framework

IM Platform (1)

Data storage:• temporal aspect (versions)

Organised data:• Fast and easy access to content• Easy processing distribution (Big Data)

Several views on same data:• Raw, extracted and/or analysed

Takes care of data replication: • No (W)ARC synchronisation required

IM Platform (2)

Extensive characterisation and data mining actions:•Process and reprocess information any time depending on needs/requests– Extract information such as MIME type, text resources,

images metadata, etc.

SCAlable Preservation Environment (SCAPE)

QA/Preservation challenges?•Growing size of web archives•Ephemeral and heterogenous content•Costly tools/actions

Develop scalable quality assurance toolsEnhance existing characterisation tools

Visual automated QA: Pagelizer

• Visual and structural comparison tool developped by the UPMC as part of SCAPE

• Trained and enhanced through a collaboration with IMF

• Wrapped by IMF team to be used at large scale within its platform Allows comparison of two web

pages snapshots Provides a similarity score as

an output

Visual automated QA: Pagelizer

• Tested on 13 000 pairs of URLs (Firefox & Opera)• 75% of correct assessment• Whole workflow runs for around 4 seconds/pair• 2 seconds for screenshot (depends on page

rendered)• 2 seconds for comparison

• Performance already cut per 2 since initial tests (map reduce)

Next stepsImprovements are to be made:•Performance •Robustness •Correctness

New test in progress on a large scale crawl:•Results to be disseminated to the community through the SCAPE project and through on-site demos (contact IMF)!

Thank you.Any questions?

http://internetmemory.org - http://archivethe.netflorent.carpentier@internetmemory.orgleila.medjkoune@internetmemory.org