Noname manuscript No.(will be inserted by the editor)

ArchiveWeb: Collaboratively Extending and Exploring WebArchive CollectionsHow would you like to work with your collections?

Zeon Trevor Fernando · Ivana Marenzi · Wolfgang Nejdl

Received: date / Accepted: date

Abstract Curated web archive collections contain fo-cused digital content which is collected by archivingorganizations, groups, and individuals to provide a rep-resentative sample covering specific topics and eventsto preserve them for future exploration and analysis.In this paper, we discuss how to best support collabo-rative construction and exploration of these collectionsthrough the ArchiveWeb system. ArchiveWeb has beendeveloped using an iterative evaluation-driven design-based research approach, with considerable user feed-back at all stages.

The first part of this paper describes the impor-tant insights we gained from our initial requirementsengineering phase during the first year of the projectand the main functionalities of the current ArchiveWebsystem for searching, constructing, exploring, and dis-cussing web archive collections. The second part sum-marizes the feedback we received on this version fromarchiving organizations and libraries, as well as our cor-responding plans for improving and extending the sys-tem for the next release.

Keywords Working with web archives, Collaborativesearch and exploration, Web archive requirements andevaluation

1 Introduction

The web is becoming an important corpus for studyinghuman society by researchers in the humanities, socialsciences, and computer sciences alike. Web archives col-lect, preserve, and provide ongoing access to ephemeral

Zeon Trevor Fernando · Ivana Marenzi · Wolfgang NejdlE-mail: {fernando,marenzi,nejdl}@L3S.deL3S Research Center, Hannover, Germany

web pages and hence encode important traces of hu-man thought, activity, and history. Curated web archivecollections contain focused digital content from archiv-ing organizations, groups, and individuals related tospecific topics or covering specific events, which arecollected to provide representative samples and pre-serve them for future exploration and analysis. How-ever, there have been only a few concerted efforts toprovide tools and platforms for exploring and workingwith such web archives.

This article focuses on the ArchiveWeb1 platformwhich provides facilities to collaboratively explore andwork with web collections in an interactive and user-friendly way, both for research and for learning. Inparticular we focus on supporting collaborative explo-ration and analytics through a user friendly searchingand sharing interface (ArchiveWeb). Our goal is to al-low users (e.g., archivists and librarians, web archivecurators, researchers) to pose queries in the contextof a larger exploration process where search resultsare stored for sharing with colleagues and collabora-tors, later discussion, and analysis [13]. This enablesthe user to consider dependencies between queries in-dependent of the order in which they are posed, therebyregarding them in the context of later queries andquery results. Building on previous work we did for col-laborative learning environments [13], we enable usersto build their collections incrementally and collabora-tively, which is an important step to support their re-search work. We also allow researchers and their col-laborators to approve, add, or remove results to theircollections and to discuss their significance.

1 http://archiveweb.l3s.uni-hannover.de/aw/index.jsf

arX

iv:1

702.

0043

6v1

[cs

.DL

] 1

Feb

201

7

2 Zeon Trevor Fernando et al.

In the following section, we provide a short overviewof related work, in Sect. 3, we describe a life cycle modelfor web archiving, and in Sect. 4, we give more detailsabout the analysis of user requirements on which webased the ArchiveWeb interface. In Sect. 5, we describetwo sample web archive collections we are working with.Section 6 includes a description of the main functional-ities of our system, and Sect. 7 presents the evaluationdesign with a detailed discussion of the evaluation re-sults. In Sect. 8, we summarize future extensions for thenext ArchiveWeb release and finally draw conclusionsin Sect. 9.

2 Related Work

Information retrieval (IR) models and algorithms havebeen extremely successful during the last 20 yearsin providing everybody with easy access to the vastamount of information available on and through theweb. Web and IR conferences connect a large num-ber of researchers working on problems related to websearch, the Social Web and data mining, and relatedtopics. Surprisingly little work has been spent, how-ever, on issues related to temporal retrieval, or explo-ration and analysis of large temporal collections likeweb archives [4], even though the need for focusing re-search on these issues has been recognized [1,18].

The Alexandria project2 focuses on open questionswhere radical changes and advances in the state ofthe art are necessary to move ahead in our abilitiesof indexing, retrieving, and exploring past web con-tent. Models and algorithms for temporal informationretrieval developed in the context of this project willtake the unique temporal dimension of web archivesinto account. Semantic entity-based indexing will sup-port exploration of temporal web content and evolvingentities in a more user-oriented way than conventionaldocument-based retrieval. Finally, user input gatheredby complex and collaborative search and analysis pro-cesses of archive users will further enable us to im-prove web archive indexing and enrichment consider-ably. While this project targets web archives, similarissues arise in any digital archive, and solutions devel-oped within Alexandria will likely be applicable to othertypes of archives as well.

Often desired search functionalities in web archivesinclude full-text search with good ranking, followed byURL search [16]. A recent survey showed that 89%of web archives provide URL search access, and 79%give metadata search functionalities [7]. Some existing

2 http://alexandria-project.eu/

projects that provide limited support for web archiveresearch are discussed below.

The Wayback Machine3 is a web archive accesstool supported by the Internet Archive. It providesthe ability to retrieve and access web pages storedin a web archive through URL search. The resultsfor each URL are displayed in a calendar view whichdisplays the number of times the URL was crawledby the Internet Archive web crawlers. Archive-It andArchiveTheNet4 are web archive services provided bythe Internet Archive and the Internet Memory Foun-dation. These services enable focused archiving of webcontents by organizations, such as universities or li-braries, that otherwise could not manage their ownarchives. The Memento Project5 enables the discoveryof archived content from across multiple web archivesvia URL search.

A few researchers have worked on providing new in-terfaces and visualizations for searching, exploring, anddiscovering insights from web archives. Odijk et al. [14]present an exploratory search interface to improve ac-cessibility of digital archived collections for humani-ties scholars, in order to highlight different perspec-tives across heterogeneous historical collections. Themotivation for this work derives from the huge amountof digital material that has become available to studyour recent history, including books, newspapers, andweb pages, all of which provide different perspectiveson people, places, and events over time. The authorsconnect heterogeneous digital collections through thetemporal references found in the documents as wellas their textual content, in order to support scholarsto detect, visualize, and explore materials from differ-ent perspectives. Padia et al. [15] provided an overviewof a web archive collection by highlighting the collec-tion’s underlying characteristics using different visual-izations of image plots, wordle, bubble charts, and time-lines. Lin et al. [10] present an interactive visualizationbased on topic models for exploring archived content,so that users can get an overview of the collection con-tent. The visualization displays a person-by-topic ma-trix that shows the association between US senators’websites and the derived topics. The interface also pro-vides drill-down capabilities for users to examine thepages in which a topic is prevalent.

All of the above tools and interfaces help supportthe exploration and search of web archives for indi-vidual users and researchers. In addition, ArchiveWebaims at supporting the collaborative exploration ofweb archives. Previous research on helping users keep

3 http://archive.org/web/4 http://archivethe.net/5 http://timetravel.mementoweb.org

ArchiveWeb: Collaboratively Extending and Exploring Web Archive Collections 3

track of their resources includes tools that provide bet-ter search and organizational facilities based on meta-data/time [5] or tagging [3]. Our system provides sim-ilar organizational functionalities refined through closecollaboration with several learning communities andprevious work on the LearnWeb platform [13], thusgaining advantage from several years of developmentand user feedback in that context. ArchiveWeb buildson the LearnWeb experience which already supportscollaborative sensemaking by allowing users to shareand collaboratively work on resources retrieved fromvarious web sources.

LearnWeb6 is a learning and competence develop-ment environment, which allows users to share and col-laboratively work on resources collected from the webor user-generated [13]. It provides users with a searchinterface for resource discovery and sharing across var-ious Web 2.0 services such as YouTube, Flickr, andSlideshare, including LearnWeb itself, and offers a Per-sonal Web 2.0 Learning Space. In order to support col-laborative searching, LearnWeb provides automatic re-source annotation. Resources in LearnWeb can be book-marked, tagged, rated, and discussed by all users whoare allowed to access them. Comments on particularlearning resources can be used to enrich the description.Users can create folders to bundle resources that be-long to the same learning context. Hence, the LearnWebcommunity can collaboratively identify the best learn-ing resources for specific learning domains. A discussionof the full potentialities and affordances of LearnWeb asa collaborative platform is beyond the scope of this pa-per, but can be found in a series of published studies [11,12,13].

The Integrated Digital Event Archive and Library(IDEAL) system7 addresses the integration of digitallibraries and archival technologies in support of stake-holders interested in studying important events. It fo-cuses on events falling into two broad categories: (1)related to crises or tragedies as well as recovery and(2) government/community-related events (e.g., elec-tions, demonstrations). The system monitors web-based(news, government, and other websites) and social me-dia activity (tweets) to automatically detect interest-ing events and respond to general and specific requestsarchiving requests as well. Once events are identified,intelligent focused crawling and filtering approaches areused to ingest content and generate collections withhigh precision, recall, and low bias as is needed forscholarly study by social scientists. The system also in-corporates a wide range of integrated services such as

6 http://learnweb.l3s.uni-hannover.de7 http://www.nsf.gov/awardsearch/showAward?AWD_ID=

1319578

browsing, searching, recommendation, clustering, senti-ment analysis, summarization and visualization of data,information, or context.

The Big UK Domain Data for the Arts and Hu-manities (BUDDAH) project8 aimed to develop a the-oretical and methodological framework within which tostudy the archived UK web and web archives in gen-eral [19]. To demonstrate the value of web archivesto humanities researchers and to build a community,a wide range of research projects was undertaken -from analyzing disability action groups online to study-ing the Ministry of Defense’s recruitment strategy. Theproject also worked on developing a suite of tools tosupport the analysis of the UK web archive and intro-duced an exploratory search interface SHINE9 throughwhich humanities scholars and social scientists accessedweb archives [8].

The RESAW network,10 established in 2012, aimsto establish and operate a collaborative world-classtransnational European research infrastructure to en-able cross-border studies of the archived web by inte-grating and opening up existing national web archives.It will facilitate easy access to large amounts of cul-tural heritage big data, as well as provide searching,selecting, and analysis of the material itself, thus mak-ing the research process more efficient and enhancingthe European research area. At the moment, RESAWis a network of scholars and libraries interested in WebArchives and a research proposal to which we are con-tributing. We intend to make ArchiveWeb available tothe participating libraries, both to get additional re-quirements from our partners as well as to supportthe collection building process for various national webarchives.

3 The Web Archiving Life Cycle Model

Despite the increasing number of web archiving pro-grams, best practices and a common methodology forweb archiving have not been established yet [2]. TheWeb Archiving Life Cycle Model (WALCM), developedby experts at the Internet Archive, suggests a com-mon framework for organizations seeking to archive theweb. The model, reproduced in Fig. 1, includes sev-eral phases representing the various steps of a commonworkflow which organizations can refer to in order tocreate or improve their web archiving programs.

The original model is focused on the institutionalpolicy collection; target users are archiving institutions

8 http://buddah.projects.history.ac.uk/9 https://www.webarchive.org.uk/shine

10 http://resaw.eu/about/

4 Zeon Trevor Fernando et al.

and curators of such collections. Each phase of this pro-cess is supported by the leading web archiving serviceArchive-It (mentioned above).

Fig. 1: Web Archiving Life Cycle Model

In the first phase of our work, we referred to the orig-inal WALCM model to prepare the investigation of cur-rent practices and of user requirements (Sect. 4). Figure1 shows the steps of the WALCM model highlighted inred which we used as a reference to guide the semi-structured interviews and to analyze the results. Wefocused on three steps related to high-level policy de-cisions (i.e., Vision and Objectives, Access/Use/Reuse,Preservation), and on three day-to-day tasks of the webarchiving process (i.e., Appraisal and Selection, Storageand Organization, Quality Assurance and Analysis).

As regards the curatorial activities of storage andorganization, and quality assurance and analysis, ourinterpretation is slightly different from the original one,though. With storage and organization we do not re-fer to the temporary or long-term storage plan of theorganization, but rather we consider the possibilityfor ArchiveWeb users to organize resources in sub-collections. Quality assurance and analysis is based onthe possibility for ArchiveWeb users to collaborativelyannotate and comment on resources in order to stateor to discuss their relevance.

4 User Requirements Analysis 2015

4.1 Motivation and Setup

The motivation of our study was to provide a user-friendly interface to allow various kinds of users to ac-cess archived web collections and use the resources invarious scenarios, e.g., for research or for learning.

In order to do so, in Summer 2015 we carried out apreliminary analysis of user requirements [6]. Partic-ipants were colleagues from the Internet Archive, aswell as from libraries and similar institutions who arecreating/curating Archive-It collections, e.g., StanfordUniversity Libraries, Cornell University, University ofToronto, Columbia University, the National Museum ofWomen in the Arts, and the New York Arts ResourcesConsortium. The purpose of the study was to collectand analyze requirements for building and using webarchive collections to be used as important input fornew algorithms and tools to improve web archives. Theultimate goals of our work are to

– understand what kinds of web archive collections arebeing curated by different kinds of organizations

– contribute further improvements for working withweb archive collections, especially regarding inter-face and search functionalities

With reference to the web archiving life cycle (Fig.1), our interest was focused on specific steps relatedto policy decisions (e.g., Vision and Objectives, Ac-cess/Use/Reuse, Preservation), and data management(e.g., Appraisal and Selection, Storage and Organiza-tion, Quality Assurance and Analysis). Based on thesesix main categories, we used five leading questions toguide the semi-structured interviews:

1. What collections are built and what is the motiva-tion for building these collections?

2. Who decides which collection is built and who isresponsible for selecting which resources go into aspecific collection?

3. Who are/could be potential users of a collection?Do you track the use of your collection?

4. Which functionalities are important for users whowork with these collections when accessing WebArchive collections?

5. Are there any user interface improvements whichyou and your users already have on their wish list?

Respondents were free to add information and detailsrelated to their specific role/experience in the archivingprocess.

Interviews were conducted between August 3 and 6,2015, during a research visit of the investigators in San

ArchiveWeb: Collaboratively Extending and Exploring Web Archive Collections 5

Francisco. We interviewed eight experts playing differ-ent roles and taking care of various activities in thearchiving process, including the director of the Webarchiving program at the Internet Archive, three Webarchiving coordinators of University libraries, two Webarchiving coordinators in national museums, one digi-tal archivist, and one head of metadata services. Theexperts were selected based on suggestions provided bythe Internet Archive. Three interviews were conductedin person at the Internet Archive premises; the remain-ing five were scheduled online according to the availabil-ity of the experts. Interviews took on average half anhour each, and the discussions were recorded for dataanalysis purposes. We used a consent form to informthe participants on the procedures and protection pre-cautions and to collect their signatures for agreement.

Participation in the project was voluntary, and any-body was free to withdraw from the project at anypoint. Personal information is treated confidentially:Name and affiliations are anonymized in any resultingpublications, unless participants give us explicit consentto identify them as a subject.

All recordings have been transcribed by one ofthe investigators, and qualitative content analysis ofthe transcripts has been carried out manually by twocoders. Answers have been summarized referring to thecategories in the original WALCM model (Fig. 1). InTable 1, we report a summary of the answers referringto the categories in the original WALCM model.

On the basis of this preliminary study, and buildingon previous work we did for collaborative learning en-vironments [13], we designed and built the ArchiveWebsystem to support collaborative creation and enrich-ment of web archive collections, with a focus on userinterface and searching/sharing functionalities. We in-gested 200 web archive collections from Archive-It, andasked our archiving partners for their input. In Novem-ber 2015, we prepared a Memorandum of Understand-ing (MoU) to describe how we integrate/use collectionsin ArchiveWeb, and we shared it with all participat-ing institutions to collect their official agreement andsignatures.

In the following, we briefly talk about some examplecollections, and in the next two sections, we describethe main ArchiveWeb functionalities and the evaluationprovided by the experts who participated in the usersurvey.

5 Web Archive Collections

Archive-It is a subscription web archiving service fromthe Internet Archive that helps organizations to har-

vest, build, and preserve collections of digital content.11

It was first deployed in 2006 and is widely used as aservice to collect, catalog, and manage collections ofarchived web content. Full-text search is also available,even though effective ranking is still an open issue. Allcontent is hosted and stored at the Internet Archivedata centers.

Currently, the Archive-It system is mainly used bylibrarians and curators in order to build their collec-tions. Less support is given to users and domain ex-perts who actually want to work with the collections,or to the general public. ArchiveWeb supports search-ing, collecting, exploring, and discussing web archivecollections such as the ones provided through the webarchiving service of the Internet Archive.

Currently, about 200 collections from Archive-Ithave been integrated into the ArchiveWeb system withfull metadata indexing, and can be explored througha visual interface. The Archive-It collections that wereselected consist of those curated by organizations of theexperts we interviewed as part of our user requirementsanalysis and those that cover a diverse range of topicssuch as Humanities, Arts, Society, Culture and GlobalEvents.

In order to give a representative example of the di-versity of the archived materials, we describe two spe-cific collections.

5.1 Human Rights Web Archive

The collection Human RightsWeb Archive12 (HRWA) byColumbia University Librariesis made up of searchablearchived copies of websitesrelated to human rights createdby various non-governmentalorganizations, national humanrights institutions, tribunals,and individuals. The collectionwas started in 2008 and is still

being continued, adding new websites on a regularbasis.

Identification of websites for archiving is done bysubject specialists with expertise in human rights anddifferent regions of the world at Columbia UniversityLibraries. Public nominations are provided by humanrights researchers, advocates, organizations, and indi-viduals who are involved in the creation of humanrights-related websites. Priority is given to websites

11 https://archive-it.org/learn-more/12 https://hrwa.cul.columbia.edu/about

6 Zeon Trevor Fernando et al.

Theme AnswersVision and ObjectivesWhat collections are built and whatis the motivation for building thesecollections?

In most cases the motivation for starting the archiving of digital collections has beento preserve:

– institutional or government websites– special collections received as donations of materials from individuals or orga-

nizations– event-based collections (e.g., Olympic games or election campaigns)– research outputs (e.g., faculty members hosting their funded projects including

related assets such as documents and research datasets)

Larger institutions have a more structured approach including, in some cases, fo-cused groups with their staff and researchers who determine which areas are in focuswith respect to priorities. For smaller institutions starting to archive online contentwas also seen as a challenge

Access/Use/ReuseWho are/could be potential users ofa collection? Do you track the useof your collection?

In most cases users are members of the library or of the university (e.g., professorsor researchers).For specific collections, most of the requests to access archived web collections comefrom people who are looking for documents that have been removed from a govern-ment website.Very few institutions and libraries track what users do using basic metrics (e.g.,number of hits on a collection level).

PreservationWhat kind of websites should bepreserved?

As regards to preservation, the priorities are to:

– collect non-governmental organizations websites as opposed to the governmentwebsites

– preserve resources which are in danger of disappearing from the web

Appraisal and SelectionWho decides which collection isbuilt and who is responsible forselecting which resources go into aspecific collection?

All respondents confirmed that the current efforts at their institutions are mainly in-dividual curators or subject specialists who are responsible for individual collectionsor individual archiving requests

– in some cases a collection development executive group is appointed– few institutions are collecting nominations from experts, researchers, library and

curatorial staff

Digital collections at larger institutions are growing incrementally, starting for ex-ample from collecting the institutional memory, and including websites of singledepartments and faculties, student organization websites, or special collections onspecific areas of interest. Institutions dealing with art (e.g., museums and galleries)are particularly concerned with preserving resources which are in danger of disap-pearing from the web.

Storage and OrganizationWhich functionalities are importantfor users who work with thesecollections when accessing WebArchive collections?

For curatorsCollaboration is an important aspect in order to avoid duplicating resources andefforts. Some curators are trying to find ways of collaborating in terms of

– collecting and displaying information– creating and sharing metadata schema– sharing folksonomies and tagging

Other useful functionalities would be:

– limiting the content to a certain number of seeds– downloading metadata– providing solutions to archive dynamic web pages

For final users

– Fulltext search– Annotation– Presentational functionalities (e.g., topic modeling, thumbnails of the major

versions of the homepage per seed, link labels analysis

Quality Assurances and AnalysisAre there any user interfaceimprovements which you and yourusers already have on their wishlist?

Improvement of the interface

– Longitudinal data exploration– query of subcollections– integration of web archive resources into the general digital environment

Table 1: User Requirements Analysis - Summary of responses

ArchiveWeb: Collaboratively Extending and Exploring Web Archive Collections 7

hosted in countries that do not have any systematicweb archiving initiatives in place. Websites of intergov-ernmental organizations such as the United Nations arenot included in the collection.

Archive-It services are used to maintain the collec-tion, and the Internet Archive and Columbia Univer-sity Libraries store copies of the resulting data. Thecollection includes over 711 websites with more than50 million searchable documents and over 115 millionarchived documents with an archived data size of morethan 5TB.

The HRWA collection provides a good balance be-tween websites of large and well-known human rightsorganizations based in North America and Europe andwebsites of smaller organizations from other regions.One example of a website that no longer has a live ver-sion but can still be accessed by researchers via the Webarchive collection is TibetInfoNet, which monitored thesituation in Tibet http://www.tibetinfonet.net/.This page has been captured between May 2008 andJuly 2015.

The ArchiveWeb screenshot shows the human rightsgroup restricted to “Tibet” -related resources, with theTibetInfoNet resource on the right side (Fig. 2). TheURL provided in ArchiveWeb below the screenshot onthe right side points to the original website, which is nolonger available on the live web, but can be retrievedin its previous versions through the Wayback Machine(as described in Sect. 6.4).

5.2 Contemporary Women Artists on the Web

The collection Contemporary Women Artists on theWeb13 has been collected by the National Museum ofWomen in the Arts (NMWA) since December 2011.The NMWA is a gender-specific museum, located inWashington, and is the only museum solely dedicatedto celebrating women’s achievements in the visual, per-forming, and literary arts. Since opening its doors in1987, the museum has acquired a collection of morethan 4,500 paintings, sculptures, and works on paperand decorative art.

The collection Contemporary Women Artists on theWeb includes three components: (1) Individual websitescreated by artists working in conceptual or new mediaart, (2) Artist profiles of women artists represented bycontemporary art galleries, and (3) Women artist orga-nizations.

One representative example of a resource in this col-lection is http://jodi.org, see Fig. 3, which has been

13 https://archive-it.org/collections/2973

Fig. 3: Jodi Art Collective

captured 1418 times between April 16, 2009, and March31, 2016, and which includes 726 videos.

As described in Wikipedia,14 it is the result of theinitiative of two authors who work together toward ashared aim: Joan Heemskerk (born in 1968 in Kaat-sheuvel, the Netherlands) and Dirk Paesmans (bornin 1965 in Brussels, Belgium). Their background is inphotography and video art. Since the mid-1990s, theystarted to create original artworks for the World WideWeb, and a few years later, they turned to software artand began the practice of modifying old video games tocreate a new set of art games [17].

Opening windows cascade across the screen, errormessages squawk, and files replicate themselves end-lessly. As graphics explode across the screen, the viewergradually realizes that what had initially appeared tobe a computer glitch is in reality the work of an ir-rational, playful, or crazed human [9]. Webpages are

14 https://en.wikipedia.org/wiki/Jodi_(art_collective)

8 Zeon Trevor Fernando et al.

Fig. 2: Human Rights Example Resource

interactive, and a new perspective is displayed as theuser clicks on various elements on the page.

Each day the webpage http://jodi.org changes,and a new piece of art is displayed. If the resource wasnot archived, all work would be lost. For this reason, itis important to archive at least a capture once a day inorder to document the work of these artists over time.

6 ArchiveWeb System: Main Functionalities

Fig. 4: The ArchiveWeb User Model

With the requirements described in Sect. 4 and thecollections sketched in Sect. 5 in mind, we worked onproviding the ArchiveWeb system to web archivists andlibraries to facilitate collaborative exploration of multi-ple focused web archive collections.

At the design level, in ArchiveWeb we did not wantto replicate services offered by the Archive-It servicesuch as scoping, data capture, or risk management.Rather than focusing on the institution and collectionlevel, we focus more on the final user who wants to ex-plore and work with web archive collections. Thus, Fig.4 highlights a more detailed inner circle with focus onthe Web Archive User (instead of on the collections).Besides the curatorial tasks already present in the origi-nal WALCM model (i.e., Appraisal and Selection, Stor-age and Organization, Quality Assurance and Analysis)we detail four activities related to collaborative searchand exploration of resources, as well as organizationalfunctionalities (i.e., Nomination, Search, Organization,and Collaboration).

We used these categories, highlighted in red in themodel, to guide the design of the system functionalities,as well as the evaluation of the system in 2016 (Sect.7).

The main features provided by ArchiveWeb to sup-port this exploration are: search - searching across mul-tiple collections in conjunction with the live web, orga-nization - grouping of resources for creating, mergingor expanding collections from existing ones and collab-oration - enrichment of resources in existing collectionsusing comments and tags for the purpose of discus-sion. The following sections give a detailed descriptionof these main system functionalities. In Sect. 6.1 we

ArchiveWeb: Collaboratively Extending and Exploring Web Archive Collections 9

describe the technical workflow of the system, first de-scribing how the data were collected and then how thesystem was implemented.

6.1 Technical Workflow

Data Collection Process. The Archive-It collection datawas obtained by scraping the Archive-It website15. Weselected a subset of collections to be crawled based onthe subject metadata field that included the topics: So-ciety & Culture, Arts & Humanities, Universities & Li-braries. Also collections specifically curated by the eval-uators institutions were included in the crawl.

For each Archive-It collection, we collect and storethe following metadata: collection title, collecting in-stitution, description, subject, and collectors. All re-sults/seed URLs displayed on each Archive-It collectionpage (e.g., Human Rights collection16) are also crawled.For each result, we store the following metadata: URL,title, description, subject, collector, creator, publisher,language, format, and type. If a result does not havemetadata for title, subject, or language in Archive-It,we fetch the title, language, and subject from the meta-keywords if available from the HTML page of the re-sult. While crawling each result of a collection, we storethe individual capture information fetched using theTimeMap component of the Memento framework. ATimeMap is a list of URIs of captures of the originalresult/URL (e.g., 4genderjustice.org17).

We update the Archive-It collections to keepArchiveWeb content up to date with the informationin the Archive-It portal by crawling and updating onlythose collections that have any result that has been cap-tured/crawled recently (3 months before current timeof crawl).

Implementation. The interface is implemented usingthe MVC framework JavaServer Faces (JSF) andPrimeFaces UI component library for JSF. Navigationacross the ArchiveWeb system is facilitated through agrid layout in order to guide and keep the user in con-text and engaged at all times. For more details aboutthe implementation and conceptual design of the initialplatform LearnWeb on which the ArchiveWeb systemhas been built, we refer the reader to [13].

In the following, we discuss the extensions that weincorporated during the first year of the project in orderto support the main features of ArchiveWeb. To provide

15 https://archive-it.org/explore/?show=Collections16 https://archive-it.org/collections/147517 https://wayback.archive-it.org/1475/timemap/link/http://4genderjustice.org/

access to the crawled Archive-It data from the above-mentioned process, they have to be modeled as digitalobjects (resources, groups/collections) and stored in arelational database format, using the MariaDB repos-itory. The data in the collections are made searchablethrough a metadata index, using Solr. Comments andtags added to resources by various collaborators whilecurating collections are also indexed in order to sup-port collaborative search and exploration. Search filtersbased on the various metadata fields of the results wereincorporated using the Solr faceted search feature tonarrow down search results.

As each of the resources/URLs has a number ofarchived versions, the challenge was how to incorporatenavigation through these versions in the interface. Tosupport users to navigate through the different versions,a timeline view similar to the Wayback Machine inter-face was incorporated (see Fig. 8). A simple list viewwas also included to provide a one-step navigation forresources with very few archived versions. Finally, wealso implemented a functionality for merging existingcollections and creating new ones, so that users have theflexibility to merge multiple collections covering similartopics/subjects and build upon them.

6.2 Search

Fig. 5: Options Dialog

ArchiveWeb provides a keyword-based search sys-tem that returns results from ArchiveWeb collectionsas well as from the live web (Fig. 6) using the BingSearch API.18 ArchiveWeb collections includes Archive-It collections as well as user created collections stored inthe system. If users search for a keyword (e.g., “humanrights”), ArchiveWeb returns a list of results from these

18 http://datamarket.azure.com/dataset/bing/search

10 Zeon Trevor Fernando et al.

Fig. 6: Search Page

sources, indicating whether the resource comes from aspecific Archive-It/ArchiveWeb collection or from thelive web (Fig. 6b). The search results are presented inthis manner, so that live web results which are not yetpart of any collection could be added to existing ones.The results being returned from Archive-It collectionsare based on a full metadata index for these collections.A full-text index of all content is planned for the nextversion. Besides webpages, images (from Bing, Flickr,Ipernity) and videos (from YouTube, TED, Yovisto,Vimeo) can also be searched, but not yet from Archive-It collections. For Archive-It collections, each search re-sult provides a pointer to the resource item saved inArchiveWeb (Fig. 6c) and displays information aboutwhen it was captured in Archive-It (Fig. 6d). Live webresults are not yet saved in ArchiveWeb but can becopied by the user into a group (Fig. 6e); in this case,the Options dialog box (Fig. 5) provides details aboutwhen/if this page has been indexed by the InternetArchive, retrieving the information from the WaybackCDX server API.19

Search results can be refined using faceted search fil-ters visible below the search box (Fig. 7). For example,for our evaluation it was important to provide filters toshow only the resources in a specific collection (Filter:Group) or that were archived by a specific institution(Filter: Collector).

19 https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server

6.3 Organization

ArchiveWeb provides the functionality to organize re-sources into collections (groups of resources) accord-ing to clearly defined and coherent themes/topics. Thisfunctionality allows working with existing groups, cre-ating new collections/groups and sub-groups, addingnew resources to groups, and moving resources betweengroups.

A group overview interface allows browsing throughexisting collections available within ArchiveWeb andthe collections that a user has created or joined.Descriptions for every collection provide informationabout the topic/theme and what kind of resources itcontains. For Archive-It collections, the description re-ports the details available on the public Archive-It in-terface along with a reference link to the specific collec-tion in the Archive-It platform. It is possible to filtercollections by searching through the title and descrip-tion fields.

After joining specific groups/collections users havethe opportunity to edit metadata of existing resourcesand to contribute new resources from other existing col-lections or from the live web. The ArchiveWeb collec-tions, derived directly from their Archive-It counter-parts, are “read only”. Users can browse through theresources of each collection using the advanced visual-ization and exploration functionalities of ArchiveWeb,but they cannot change the original collection. They

ArchiveWeb: Collaboratively Extending and Exploring Web Archive Collections 11

Fig. 7: Refinement through Faceted Search Filters

Fig. 8: Group Resources Interface

can create a copy of the entire “read-only” group oradd individual resources to their own collections by se-

lecting resources individually, as well as merge multipleexisting collections into one new collection.

12 Zeon Trevor Fernando et al.

Users can also organize their resources into sub-groups within collections in order to group resourcesthat are related to a similar subtopic. Resources can beuploaded to a collection either from the desktop or bysuggesting a URL. Each group has a specific interfacethat visualizes all thumbnails of the most recent snap-shot of each resource when it was added to the group(Fig. 8). Resources that no longer exist on the web,or do not have a redirect available, display a thumb-nail with the message “The page is no longer availableon the web” (we are currently implementing the func-tionality to upload a thumbnail of an earlier capture ofthe page taken from the Internet Archive Wayback Ma-chine). The overview interface within a group providesa summary of the activities of various members of thegroup, including the actions of newly added resources,resources which were edited or deleted, and users whohave joined/left the group.

6.4 Collaboration

After a resource is added to a ArchiveWeb collection,the resource can be enriched with additional commentsand tags (in addition to the metadata already pro-vided). The comments on a resource can be used todiscuss why this resource has been chosen as a seed fora collection, to decide upon the crawl frequency andcrawl depth, as well as any other issues and discus-sions which should be documented during the collectionbuilding process (Fig. 9). By exchanging comments, col-laborators can also discuss and decide on the relevanceof a suggested seed resource for a particular collection.20

The use of tags helps categorize or label a resource withsubjects or topics covered by the resource, making itpossible to browse collections by filtering based on cer-tain tags. Users can also edit metadata such as title,description, and author fields.

During the collection building process, multiple col-laborators can join a newly created ArchiveWeb collec-tion and suggest seed URLs from the live web or fromexisting collections in ArchiveWeb. Collaborators canadd comments and tags to these seed URLs as high-lighted above to indicate the importance of this resourceto the collection. Once the collaborators have finalizedthe seed URLs, they use the Archive-It system to cu-rate this collection by specifying the seeds with cor-responding metadata, crawl depth, and frequency. Inthe future, we will investigate a direct connection be-tween ArchiveWeb and the Archive-It system, in order20 ArchiveWeb is not (yet) directly coupled to the Archive-It administrative interface: curators have to switch to thisinterface to add suggested seed URLs to the correspondingArchive-It collection.

to provide a seamless environment for working with andarchiving Web collections.

The system allows users to archive a single page ora website (resource) by clicking on the “Archive Now ”button which sends a request to the Wayback Machineto archive it. This functionality is similar to the “SavePage Now ” feature of the Wayback machine, and itsupports users to gather captures easily as they workwithin the system. All captures from Archive-It as wellas the Wayback captures in ArchiveWeb added usingthe “Archive Now ” functionality are visualized both asa list and in a timeline view to help users navigatethrough the different archived versions that are avail-able. The timeline view visualizes a timeline displayingthe number of archived versions aggregated by monthfor each resource. Figure 8 shows a timeline view on thebottom right.

7 Evaluation and Feedback 2016

At the end of 2015, the release of ArchiveWeb, whichtook into account the initial requirements collected insummer 2015, was ready for a first evaluation. In Febru-ary 2016, we invited all experts, who provided the initialuser requirements, to participate in a task-based evalu-ation of the system. All of them agreed to participate.

For the evaluation, we imported 200 publicly ac-cessible web archive collections from Archive-It intoArchiveWeb in order to test the potential of the sys-tem to support collaborative work with such collec-tions. The evaluation was done using a task-based eval-uation design, analyzing both quantitative interactiondata (usage logs) and qualitative feedback in the formof interviews. We asked the experts to carry out twosets of tasks: individual and collaborative. The individ-ual tasks included creating a sub-collection from an ex-isting collection of their institution, enriching an exist-ing collection with additional relevant resources fromthe live web and annotating resources with informa-tion about why they should be included and how of-ten they should be archived. The collaborative tasks in-volved selecting ten featured resources from their collec-tions, including them into a joint group shared with allother evaluators, and discussing the reasons why suchresources were included. The final task involved creat-ing a new collection about a shared topic among theevaluators, searching for relevant seed URLs for thiscollection using ArchiveWeb, and agreeing with the restof the evaluators about which seeds should be included,what should be the archiving frequency, and why.

The evaluation period spanned about three weeks,after which we invited the evaluators to give us feed-back through a questionnaire and through follow-up in-

ArchiveWeb: Collaboratively Extending and Exploring Web Archive Collections 13

terviews to fully understand their experience. In the fol-lowing, we will summarize the feedback we received andpoint out specific interesting responses and suggestions,based on the questions we asked. The headings denotethe questions being asked.

7.1 Usage Log Analysis

In the ArchiveWeb system, we log the various ac-tions carried out by the user while working with thesystem. The types of actions logged include annota-tion (tags or comments added to resources), search-ing (queries issued, type of search such as image,video, webpages, search filters used), resource-level ac-tions (additions/deletions/edits) and group-level ac-tions (metadata edits/subfolders actions/search withina group/group merging).

After the evaluation, we analyzed the usage logsto understand how the evaluators worked with theArchiveWeb system and how the various tasks werecompleted. For the initial collaborative task of selectingfeatured resources from their collections to be added toa shared group with the evaluators, 6 of the 8 evaluatorsparticipated in this task. Most of the evaluators addedfew featured resources from their own collections to theshared group, but one of the evaluators added resourcesdirectly from Bing by searching for known URLs, andone evaluator added resources that were previously up-loaded to a newly created collection. All of the evalu-ators tagged their resources, and one evaluator addedcomments to discuss why a resource was included. Noneof the evaluators added comments on resources addedby others; thus there was no discussion or collaborationbetween the evaluators. This could be due to differentinterests in topics among the evaluators. Evidence fromthe usage logs shows that the evaluators could not carryout the final collaborative task of creating a new sharedcollection based on a shared topic and collecting seedURLs due to the short time period of the evaluationand limited communication between the evaluators.

As specific examples, let us here summarize the indi-vidual tasks carried out by two evaluators as observedfrom the logs. A Web archiving coordinator of a uni-versity library initially joined his institution’s variouscollections covering topics such as Human Rights, Arts& Humanities, Society & Culture. He then created anew collection by merging resources from two existingcollections. Later, he carried out 20 searches related toclimate justice or urban planning with the use of filterssuch as “author”, “tags”, “groups”. He added tags andcomments related to crawling strategies of resourcessuch as “one-time capture”. Finally, he worked for 50

min on the initial collaborative task and added 10 fea-tured resources to the joint group with comments dis-cussing their importance.

A Web archiving coordinator of a national museumcreated a new collection and added about 10 resourcesfrom an existing collection of her institution. Then shecarried out 5 searches related to photography/ photog-raphy galleries using filters such as “service” or “groups”and added 5 resources from Bing to a sub-folder cre-ated within the new collection. Later, she added tagsand comments such as “photo gallery” and “archivemonthly”. Finally, she searched for 5 known resourcesfrom Bing and added them along with 3 other resourcesfrom an existing collection to the joint group with tags,in order to get a feel of how close the workflow ofArchiveWeb was to the existing one. This final taskwas completed in 20 min.

7.2 Evaluation: Questions and Answers

As we mentioned in Sect. 4, one of the goals of our workis to contribute further improvements for working withweb archive collections, especially regarding collabora-tive search and exploration of resources, as well as orga-nizational functionalities. For this reason, we asked theevaluators to give us feedback about the current func-tionalities, as well as suggestions for further improve-ment. In general, all evaluators appreciated the curato-rial functionalities provided by the system. The answerswere analyzed manually and are categorized accordingto the four main steps reported in the ArchiveWeb UserModel in Fig. 4.

7.2.1 Nomination

Seed URL Discovery. The integration of searchingacross web archive collections as well as the live web,providing the ability to suggest additional material thatcould be archived, is a new feature which is not presentin existing systems used for curation/creation of webarchive collections, and it was much appreciated by theevaluators: I really liked the feature of searching the webin conjunction with the web archive collections availablein the system, an entirely new feature which was seam-less and straightforward.

Others positively stressed the possibility to archiveweb pages directly from the interface: I also like thatthe “Archive Now” feature is so well embedded in theplatform - it will make it very easy for people to gathernew captures as they work, without having to leave theinterface.

14 Zeon Trevor Fernando et al.

Nomination tool. Nomination for collaborative collec-tion building was mentioned as well: A nominationfunction would be nice – like URL suggestion insteadof having to upload all resources. ... The inclusion ofscreenshots and ability to pull in resources from liveweb could make this tool a better option than the UNTnomination tool 21 for collaborative collection building.... Ability for non-members to submit tags for contentor suggest content for future archiving could be submit-ted for approval first.

7.2.2 Search

A few remarks concerned our ranking of results andincluded suggestions for (i) providing a transparent ex-planation of how the relevance ranking is determined,(ii) providing the total number of captures of a pagealong with the capturing period in order to determinethe popularity of an archived result, and (iii) cues as tohow many results were returned. Users in general likedthe information provided on the results page: Yes, Ithink the highlighted search term and snippet are suffi-cient for each item in search results. I like that the webarchive hits are placed before the live web hits, but oth-ers also mentioned: The results would provide enoughinformation if the archived resources had rich enoughmetadata. It would make way more sense if there weremore tags.

A related suggestion concerned results from differ-ent sources. Users liked the possibility to retrieve re-sults from several sources: The search results seem tooffer enough information. I appreciate that the usercan filter by the source of the content (e.g., Archive-It,ArchiveWeb, Bing), but others asked for more clearlyseparating results from different sources or defaultingto particular sources: I do wonder about the utility ofhaving the Bing results mixed with the Archive-It resultsunless the idea is that the user will use these results toadd resources to collection ... I would think that the de-fault search results would just be Archive-It collectionsthen with the opportunity to expand out next.

Other remarks asked for advanced search function-alities such as limit to collection, limit to domain/path,search within title only. The experts also requested full-text search of Archive-It records (WARCs) instead ofjust metadata search.

7.2.3 Organization

Efficiency through the interface. From conversationswith Archive-It partners and the NDSA Web ArchivingSurvey, it became clear that web archiving on the whole

21 http://digital2.library.unt.edu/nomination/

takes a lot of time to do well. Therefore, for curatorialneeds, efficiency is a critical issue. This asks for makingit as quick and intuitive as possible to move from onetask to another and making tasks as efficient as possible(providing bulk operations, summary views, etc.)

One of the evaluators stated that: ArchiveWeb in-creases curatorial functionality over that of Archive-It,e.g., on-the-fly creation of groups, moving resources be-tween groups, and easily annotating resources.

Collection management. Several of the new functional-ities of ArchiveWeb were positively highlighted by theevaluators including (i) the ability to curate new arbi-trary collections of seed records from across multipleweb archive collections, (ii) the possibility of having re-sources exist at multiple levels (e.g., personal, group,sub-folder), and (iii) the ability to create collaborativecollections with colleagues from various institutions.

Improving the user interface. A few remarks werespecifically referring to curatorial functionalities, suchas bulk operations: selection of multiple resourceseither by clicking, searching, or filtering and thenadding/moving to a collection; collective tagging ofmultiple seed records; and initiating archiving for mul-tiple seed records. One of the evaluators mentioned: It’dbe great to tag similar collections and allow them to begrouped together without having to create a new collec-tion.

An undo feature would also be useful forArchiveWeb: I would like to see some sort of “undo”feature for mistakes that are made in the system (forexample, if I deleted something from a group and im-mediately felt it was in error, I could undo that action).

Exporting resources and metadata. Several expertsmissed a way for exporting resources and metadata forresearch / usage outside of ArchiveWeb, and means torequest derivative data from Archive Research Servicessuch as WATs22, WANEs23, LGAs24, etc.: I would ap-preciate means of interacting with content instead ofjust searching for web pages, e.g., ngrams, word fre-quency like UK Web Archives SHINE 25.

22 Web Archive Transformation (WAT) files contain keymetadata such as capture information, essential text and linkdata, and other information extracted from (W)ARC files.23 Web Archive Named Entities (WANE) files contain a listof people, places and organizations mentioned in each validarchived record, extracted using Stanford Named Entity Rec-ognizer.24 Longitudinal Graph Analysis (LGA) files feature a com-plete list of what URLs link to what URLs, along with atimestamp, within an entire web archive collection.25 https://www.webarchive.org.uk/shine

ArchiveWeb: Collaboratively Extending and Exploring Web Archive Collections 15

Experts liked the ability of our interface to providean easy overview over resources and collections usingscreenshots, but also suggested to provide screenshots ofdifferent captures and select a specific capture as defaultscreenshot (the default we implemented for the evalua-tion was the screenshot of the last available page).

7.2.4 Collaboration

Fig. 9: Resource annotation

The evaluators positively highlighted the useful-ness of ArchiveWeb to facilitate collaborative collectionbuilding and to make web archives more useful to re-searchers: (i) It is a really fun tool to use! ... I enjoyedusing ArchiveWeb and think it has great potential infacilitating collaborative endeavors and potentially col-lection building, (ii) I think it’s incredibly easy to use.Institutions, particularly those without a great deal ofIT support, would really benefit from having an inter-face like this to help entice users to interact with webarchive collections. One of the most exciting aspects ofthis project is the potential for users to help us buildand enrich collections through suggesting resources andannotating what we’ve already archived. I’m excited tosee how this project progresses.

Collection Enrichment. One of the new functionalitiesthat was positively highlighted by the evaluators isthe possibility to add schema-agnostic classification andother information using tags. One participant (Techni-cal Manager) asked for more metadata fields for newlyadded resources besides title, author and description, tomake it easier for curators from different institutions tocollaboratively build collections: Tagging functionalityis helpful, but not quite same as being able to define acustom field name and supply a value for that field.

Other appreciated functionalities were (i) the abilityto use notes/comments in order to highlight the moti-vations for collecting (Fig. 9): I also have not previouslybeen able to tag or annotate resources within collections(Archive-It does allow for adding notes in their new re-lease, but these are only useful as an administrator of anaccount and not to the public/user), and (ii) the displayof the capture frequency, highly useful when developingcollaborative collections across institutions.

Regarding annotations, suggestions included auto-suggestion of tags from within the same collection, op-tionally allowing tag suggestions from controlled vocab-ularies, and conducting more robust mining of the textof archived resources to suggest possible annotations,plus the possibility to include external contributors fortagging: It’d be great to turn on external tagging, orpick certain trusted external users to tag.

Finally, we asked for further suggestions on anytopic important to the users.

Many evaluators found the system rather intuitive,but had to refer to the documentation about the func-tionalities that we had provided in order to feel comfort-able using the system: (i) The PDF instructions helpedsince there are not a lot of “helpers” or “tool tips” inthe interface itself. More help features baked into theinterface would help as not all the features are well de-fined or obvious. (ii) Additional text could be useful to

16 Zeon Trevor Fernando et al.

users - I would suggest hover text, where the user holdsthe mouse over something and a description would popup, with an option to close it and not see it again, ifdesired. Also, having information about the model un-derlying the metadata (schema, ontology) could helpunderstand how this metadata could be shared withother systems.

One of the experts suggested that for users who areunfamiliar with web archives, it would be good to pro-vide some information highlighting the need and chal-lenges of web archives: I think users unfamiliar with webarchives might need some tools or modules or somethingto help better understand some of the challenges of webarchives. Otherwise, they may just think of web archivesas a “blackbox” and accept collections uncritically, un-aware of why some things work and some things don’t.That could just be something like a mouse over or anFAQ entry though.

Table 2 summarizes the results of the evaluation,with particular attention on the phases supported bythe ArchiveWeb User Model.

In the next iteration of the evaluation, we will focusspecifically on collaborative tasks, in order to clearlyunderstand the collaborative potential of the systemfor Web archiving. The planned evaluation will span 1-2 months in order to give our partners more time tocollaboratively discuss and build a collection together.

8 Future Improvements in the Next Release

Based on the suggestions provided by the evaluators,we plan to incorporate the following new functionalitiesin the next release, in addition to several smaller userinterface improvements asked for during the evaluation:

Thumbnail Visualization. We are working on a specificvisualization to display the thumbnails for different cap-tures of an individual resource/seed, where the cura-tor/user can choose how many different thumbnails hewants to view.

Search Result Aggregation. We are also working on anew search interface design that aggregates results fromdifferent sources, displaying some representative exam-ples for each source, and in addition will allow users toselect a default source for which we display the resultsin detail, i.e., using a list of results view similar to thecurrent design.

Bulk Operations. We will also work on bulk operationswhich allow users to select multiple resources by click-ing, searching, or filtering and then adding them to a

collection. The system will also support bulk editing ofresources within a collection such as collective tagging(e.g., to assign a tag to multiple seed records), as wellas tagging similar resources across multiple collectionsand then allowing the option to group them togetherinto a new collection.

Exporting Resources. We will also implement the abil-ity to export resources and metadata from collectionsfor research or usage outside of ArchiveWeb, for reg-istered users who are allowed to use this export func-tionality (usage will have to be restricted because ofpotential copyright issues).

Advanced Search. We will extend our advanced searchand faceted search options to limit search within certaincollections, specific domains, or paths, and to searchwithin title or description only.

Fulltext Search. We will also incorporate full-textsearch of Archive-It websites instead of just the meta-data, even though not all evaluators even realized thatthis functionality was not yet provided, as rich meta-data descriptions are available for many collections.Obviously, this fulltext search capability will have togo along with appropriate ranking optimizations in or-der to take into account the longitudinal character ofArchive-It collections, which usually include many cap-tures per resource which have to be considered.

Collaborative Aspects. We will allow external contribu-tors to suggest tags, which can then be approved by thecurators of the collection. The system will also supportthe possibility to invite collaborators to join collectionsfrom within ArchiveWeb.

9 Conclusions

In this paper, we discussed the ArchiveWeb systemwhich supports collaborative exploration of web archivecollections. We provided a description of the main fea-tures of the system such as (i) searching across mul-tiple collections as well as the live web, (ii) groupingof resources for creating new collections or merging ex-isting ones, as well as (iii) collaborative enrichment ofresources using comments and tags.

The system has been developed based on an itera-tive evaluation-driven design-based research approach.Starting from a platform which already supportedcollaborative search and sharing of web resources,ArchiveWeb was designed to address web archive expertusers’ requirements (librarians and curators in archiv-ing institutions), as described in Sect. 4. The resulting

ArchiveWeb: Collaboratively Extending and Exploring Web Archive Collections 17

Current Functionalities Required FunctionalitiesNomination

– Seed URL Discovery: ability to curate new arbitrary col-lections of seed records from across multiple web archivecollections

– Possibility to archive web pages directly from the interface

– A “nomination” function would be nice - like URL sug-gestion - instead of having to upload all resources

– Display the capture frequency

Search

– Combined searching of seed metadata with search engineof live web

– Dynamic effects of “List View” and “Timeline View” visu-alizations for individual seed records

– Full text search of WARC– Google as well as Bing as web search options– Advanced search options (e.g., limit to collection, limit to

domain/path, search within title only)– Provide more details about the total number of captures

of a page along with the capturing period– Means of interacting with content instead of just searching

for web pages (e.g., ngrams, word frequency)

Organization

– On-the-fly creation of groups (sub-collections)– Possibility of having resources exist at multiple levels

(e.g., personal, group, subfolder)

– Bulk operations: select multiple resources and work withall at once (e.g., copy, move, edit, group, delete, annotate,share)

– Exporting of resources and metadata for research/usageoutside of ArchiveWeb

– Provide information about the model underlying themetadata

Collaboration

– Seed collection: ability to share seed URLs across collec-tions with colleagues from various institutions

– Annotation: ability to use notes/comments in order tohighlight the motivations for collecting

– Forum for users to discuss collections

– Collection Enrichment: possibility to add schema-agnosticclassification and other information using tags

– Request collaborators within the system– Allow external contributors for tagging (could be submit-

ted for approval first)

Table 2: System Evaluation - Summary of results

ArchiveWeb system, fully functional and publicly avail-able,26 has been evaluated through a task-based eval-uation study carried out with the same experts whoparticipated in the preliminary investigation. After thequantitative analysis of the logs and the qualitativefeedback from the evaluation, we are now incorporatingnew features such as exporting resources, bulk editingoperations, new visualizations, and advanced search,which will be available in the next release of the system.

Acknowledgements We especially thank Jefferson Baileyfrom the Internet Archive who provided us with the con-tacts to his colleagues at university libraries and archivinginstitutions, and for his helpful comments during the require-ments and evaluation phase. We are also grateful to all ex-perts, who participated with enthusiasm in our evaluation,providing valuable feedback and useful suggestions to improvethe ArchiveWeb system. This work was partially funded by

26 http://archiveweb.l3s.uni-hannover.de/aw/index.jsf

the European Commission in the context of the Alexandriaproject (ERC advanced Grant No. 339233).

References

1. Alonso, O., Strötgen, J., Baeza-Yates, R., Gertz, M.:Temporal information retrieval: challenges and opportu-nities. In: Proceedings of the 1st international tempo-ral web analytics workshop (TWAW 2011) associated toWWW’11, pp. 1–8 (2011)

2. Bragg, M., Hanna, K., Donovan, L., Hukill, G., Peter-son, A.: The Web Archiving Life Cycle Model. WhitePaper (2013). Available at http://ait.blog.archive.org/files/2014/04/archiveit_life_cycle_model.pdf

3. Cutrell, E., Robbins, D., Dumais, S., Sarin, R.: Fast,Flexible Filtering with Phlat. In: Proceedings of theSIGCHI Conference on Human Factors in ComputingSystems (CHI ’06), pp. 261–270 (2006)

4. Dougherty, M., van den Heuvel, C.: Historical infras-tructures for web archiving: Annotation of ephemeralcollections for researchers and cultural heritage institu-tions (2009). Available at http://web.mit.edu/comm-forum/mit6/papers/Dougherty_Heuvel.pdf

18 Zeon Trevor Fernando et al.

5. Dumais, S., Cutrell, E., Cadiz, J., Jancke, G., Sarin, R.,Robbins, D.C.: Stuff I’ve seen: a system for personal in-formation retrieval and re-use. In: Proceedings of the26th Annual International ACM SIGIR Conference onResearch and Development in Informaion Retrieval (SI-GIR ’03), pp. 72–79 (2003)

6. Fernando, Z.T., Marenzi, I., Nejdl, W., Kalyani, R.:Archiveweb: Collaboratively Extending and ExploringWeb Archive Collections. In: Proceedings of 20th Inter-national Conference on Theory and Practice of DigitalLibraries: Research and Advanced Technology for Digi-tal Libraries (TPDL ’16), pp. 107–118 (2016)

7. Gomes, D., Miranda, J., Costa, M.: A survey on webarchiving initiatives. In: Proceedings of the 15th Inter-national Conference on Theory and Practice of DigitalLibraries: Research and Advanced Technology for Digi-tal Libraries (TPDL ’11), pp. 1045–1050 (2011)

8. Jackson, A., Lin, J., Milligan, I., Ruest, N.: Desiderata forExploratory Search Interfaces to Web Archives in Sup-port of Scholarly Activities. In: Proceedings of the 16thJoint Conference on Digital Libraries, JCDL ’16, pp. 103–106 (2016)

9. Lieser, W.: Digital Art (Art Pocket). H.F.Ullmann Pub-lishing GmbH (2009)

10. Lin, J., Gholami, M., Rao, J.: Infrastructure for support-ing exploration and discovery in web archives. In: Pro-ceedings of the 23rd International Conference on WorldWide Web (WWW ’14), pp. 851–856 (2014)

11. Marenzi, I.: Multiliteracies and e-learning2.0. In: G. Blell,R. Kupetz (eds.) Foreign Language Pedagogy, Contentand Learner Oriented, vol. 28. Peter Lang (2014)

12. Marenzi, I., Nejdl, W.: I search therefore I learn - Activeand collaborative learning in language teaching: Two casestudies, pp. 103–125. Collaborative Learning 2.0: OpenEducational Resources. IGI Global (2012)

13. Marenzi, I., Zerr, S.: Multiliteracies and active learningin CLIL - the development of LearnWeb2.0. In: IEEETransactions on Learning Technologies (TLT) (2012)

14. Odijk, D., Gârbacea, C., Schoegje, T., Hollink, L.,de Boer, V., Ribbens, K., van Ossenbruggen, J.: Support-ing Exploration of Historical Perspectives across Collec-tions. In: Proceedings of 19th International Conferenceon Theory and Practice of Digital Libraries (TPDL ’15),pp. 238–251 (2015)

15. Padia, K., AlNoamany, Y., Weigle, M.C.: VisualizingDigital Collections at Archive-It. In: Proceedings ofthe 12th ACM/IEEE-CS Joint Conference on Digital Li-braries (JCDL ’12), pp. 15–18 (2012)

16. Ras, M., van Bussel, S.: Web archiving user survey. Tech-nical report, National Library of the Netherlands (Konin-klijke Bibliotheek) (2007)

17. Stalker, P.J.: Gaming In Art: A Case Study Of TwoExamples Of The Artistic Appropriation Of ComputerGames and The Mapping Of Historical Trajectories Of’Art Games’ Versus Mainstream Computer Games. Uni-versity of Witwatersrand (2005)

18. Weikum, G., Ntarmos, N., Spaniol, M., Triantafillou, P.,Benczúr, A., Kirkpatrick, S., Rigaux, P., Williamson, M.:Longitudinal Analytics on Web Archive Data: It’s AboutTime! In: Proceedings of the 5th biennial Conferenceon Innovative Data Systems Research (CIDR), Asilomar,CA, USA, January 9-12, pp. 199–202 (2011)

19. Winters, J.: Tackling complexity in humanities big data:From parliamentary proceedings to the archived web. InTuro Hiltunen, Joe McVeigh and Tanja Säily (eds.), Bigand Rich Data in English Corpus Linguistics: Methodsand Explorations. Studies in Variation, Contacts and

Change in English. Helsinki: VARIENG. (Forthcoming2017)