multilingual digital libraries: a review of issues in system-centered and user-centered studies,...

Multilingual Digissues in system

rm

ki a,

ence aeeceence andos,

information access;

studies;

ed the rules of sys-rized based on theirntered studies. Thehe system-centered

studies and four for the user-centered studies. Additional categorization was also provided ac-

d mainly focuses onmeet the challenges

raised for enabling multilingual information retrieval in Digital Libraries. Unfortunately, theseparently working inent tools and tech-lore users behaviorAs a result, further

esearch conductedIt organizes the vastspecific information.ictions from 1997 to

2012. 2013 Published by Elsevier Ltd.

* Corresponding author.E-mail addresses: [email protected] (E. Vassilakaki), [email protected], [email protected] (E. Garoufallou).

+ MODEL

1057-2317/$ - see front matter 2013 Published by Elsevier Ltd.http://dx.doi.org/10.1016/j.iilr.2013.07.002

Available online at www.sciencedirect.com

ScienceDirect

w.elsevier .com/locate/ i i l r

The International Information & Library Review (2013) xx, 1e17System design;User behavior;Literature review

efforts undertaken by a small number of researchers or research groups apisolation and therefore resulting in the development of numerous differniques. Relatively few studies have focused on the user and aimed to expand expectations when interacting with Multilingual Digital Libraries.research is needed to reach to some tangible and usable findings.Originality/value: This literature review captures the diversity of the rregarding multilingual information access and retrieval in Digital Libraries.literature in comprehensive themes and sub-themes enabling easy access toLimitations: This study reviews only papers in English due to language restrSystem-centeredstudies;User-centered

cording to type of publication.Findings: The literature concerning Multilingual Digital Libraries is vast antwo aspects the System and the Users. The majority of papers tried toMultilingual digitallibraries;Multilingualinformation retrieval;Multilingual

ature on Multilingual Digital Libraries in English language from 1997 to 20Design/methodology/approach: The present literature review has followtematic review. In particular, the identified relevant papers were categoexpressed aim on two core themes, that of system-centered and user-ceassigned papers were further analyzed and six sub-themes emerged for tKEYWORDS Abstract Purpose: This study aims to identify, collect and critical review the research liter-12.studies, infobehavior

Evgenia Vassilaka

aDepartment of Library SciAthens, Aigaleo, Athens, GrbDepartment of Library SciInstitute of Thessaloniki, SiPlease cite this article in press as: Vaand user-centered studies, informatdx.doi.org/10.1016/j.iilr.2013.07.002ital Libraries: A review of-centered and user-centeredation retrieval and user

Emmanouel Garoufallou b,*

nd Information Systems, Technological Educational Institute of

nd Information Systems, Alexander Technological EducationalThessaloniki, Greecejournal homepage: wwssilakaki, E., & Garoufallou, E., Multilingual Digital Libraries: A review of issues in system-centeredion retrieval and user behavior, The International Information & Library Review (2013), http://

tion retrieval and translingual information access both

2 E. Vassilakaki, E. Garoufallou

+ MODELTripathi, 2007). type of publication (see Annex A, Table 3). In this context, aIntroduction

Electronic information has always been produced in a va-riety of different languages but for many years its collec-tion, management and dissemination has been treatedseparately, for each language. A Digital Library was, and toan extent remains, the main body used to organize andaccess electronic information. A Digital Library is defined asan extension and enhancement of information storage andretrieval system that manipulates digital data in any me-dium (texts, images, sounds; static or dynamic images) andexists in distributed networks (Borgman, 2000, p. 42).However, until recently Digital Libraries enabled retrievalof information in one language at a time.

The Multilingual Digital Library (MDL) is a recentdevelopment and it is defined as a Digital Library that hasall functions implemented simultaneously in as manylanguages as desired and whose search and retrievalfunctions are language independent (Pavani, 2001, p.73). Multilingual Digital Libraries aim to support Multilin-gual Information Access and Retrieval. In this context, auser can search for relevant information using searchcriteria in a chosen language (preferably in his/her mothertongue) and retrieve all the documents which match thosecriteria in a unified list, regardless of the language of thedocuments or its indexing (Chen & Gey, 2004; Jorna &Davies, 2001).

The growing interest in providing access to multilingualinformation in any format led to the extension of thetraditional information retrieval techniques and develop-ment of systems to support multilingual informationretrieval and access (Yang, Lee, & Chen, 2009). Research inthis field can be traced back in 1970s with the work ofSalton (1969, 1973) but has become a major research focusduring the last decade (Jones, 2002). A significant numberof papers were published summarizing efforts in varioussub-fields of multilingual information retrieval and DigitalLibraries such as: a) natural language processing(Steinberger, 2011), b) log analysis (Agosti, Crivellari, & DiNunzio, 2011; Di Nunzio, Leveling, & Mandl, 2011), c)interoperability (Fox & Marchionini, 1998), d) multimediaretrieval and information extraction (Peters, 2012), e)interface design (Peters & Picchi, 1997), f) knowledgemanagement concepts and technologies (Chen, 2004b) toname just a few.

This paper reviews the research literature on Multilin-gual Digital Libraries in English from 1997 to 2012. Previousstudies have reviewed some of the relevant literature. Inparticular, Agosti, Di Nunzio, Ferro, and Peters (2007)provided an overview of the evolution of the Cross Lan-guage Evaluation Forum from 2000 to 2007; Mizera-Pietraszko (2007) reviewed the cross-language informationretrieval systems of some Digital Libraries whereas Erich,Coman, and Trziman (2010) focused on specific Digital Li-brary initiatives in Romania. Finally, Tripathi and Tripathi(2007) provided an overview of current status of DigitalLibraries in India. However, these studies were either notfocused on MDLs (see Agosti, Di Nunzio et al., 2007), orreviewed studies in specific regions such as Romania, India(see Mizera-Pietraszko, 2007, Erich et al., 2010; Tripathi &Please cite this article in press as: Vassilakaki, E., & Garoufallou, E., Mand user-centered studies, information retrieval and user behavior,dx.doi.org/10.1016/j.iilr.2013.07.002alone and in conjunction were run on ACM Digital Library,Library, Information Science & Technology Abstracts(LISTA), Library & Information Science Abstracts (LISA),Citeseer, Google Scholar, e-prints in Library & InformationScience (e-LiS), Digital Library of Information Science &Technology (DLIST). It should be noted that when a relevantpaper was identified, its references were examined toidentify any additional relevant papers. The searches wereconducted between August to October 2012.

The identified literature was further analyzed and twocore themes emerged based on each papers expressed aimand objectives. These were system-centered studies anduser-centered studies. Once the papers were assigned tothese two core themes, a second analysis was conducted tofurther categorize each theme. In this context, six sub-themes emerged for the system-centered studies whereasfour for the user-centered studies theme again based oneach papers expressed aim and objectives. Each paper wasthen assigned to the relevant theme creating a table (seeAnnex A, Table 4). It is worth mentioning that some papershad more than one aim and/or objectives and as such fell inmore than one theme. In these cases, the paper wasassigned to all the relevant themes.

Additional categorization was also provided according toA few studies reviewed the way multilingualism wastreated in Digital Libraries. In particular, Borgman (1997)focused on recording the multi-lingual issues involved indesigning Digital Libraries whereas Haddouti (1999)reviewed the state of the art in multilingual text retrievaland access. In addition, Chen (2004a, 2004c) provided anoverview of multilingual information access in Digital Li-braries whereas Liew (2009) documented the researchconducted for Digital Libraries from 1997 to 2007. He (2012)reviewed the outcomes of a practical session regardingachievements concerning services, users, the collectiondevelopment, and the supporting technology for enablingmultilingual information retrieval in Digital Libraries. Oneof the most recent literature reviews was that of Diekema(2012) but she focused on the core set of literature refer-ring to Multilingual Digital Libraries.

This paper is structured as follows: Firstly, the meth-odological approach undertaken for conducting the sys-tematic review is outlined. The identified themes regardingMultilingual Digital Libraries literature are described with athorough and critical discussion of papers objectives.Finally, conclusions are presented accompanied by recom-mendations about the directions research takes in terms ofmultilingual information retrieval and access in DigitalLibraries.

Methodology

The present literature review has followed the rules ofsystematic review (Hemingway & Brereton, 2009). Searchterms such as Multilingual Digital Librar*, multilingualinformation retrieval, multilingual information access,cross language information retrieval, cross languageinformation access, cross lingual information retrieval,cross lingual information access, translingual informa-ultilingual Digital Libraries: A review of issues in system-centeredThe International Information & Library Review (2013), http://

Multilingual Digital Libraries 3

+ MODELvariety of different types of papers were identified (e.g.Blog entry, wiki, thesis) but the majority of studies werepublished as conference proceedings or journal papers. It isworth noting that no preference for a specific conferenceor academic journal was discovered. In terms of themethods employed (see Annex A, Table 2), the majority ofpapers used the Case Study approach; but if split down tothe two core themes, Case Study was mainly adopted bythe System-centered studies whereas a mixture of quali-tative and quantitative research methods were used by theUser-centered studies (see Annex A, Table 5). The presen-tation and discussion of the literature is presented usingthe themes which emerged from the analysis.

Themes

System centered studies

Exploration and challengesThirteen system-centered studies focused on the investi-gation of the challenges that different technologies posedin enabling multilingual information retrieval in Digital Li-braries. These studies proposed either new approaches, keyfeatures, models or even repositories. In particular, Cheng,Young, and Wong (1999) investigated the way the wordsegmentation model could be cascaded. In addition, Yang,Luk, Yung, and Yen (2000) looked into boundary detectionapproaches for segmentation whereas Wang, Lu, and Chien(2004; Wang, Teng, Lu, & Chien, 2006) proposed the use ofWeb as the multilingual corpus to enable the retrieval ofinformation in multiple languages in Digital Libraries.

The isolated query translation and pseudo-translatedexpanded records approaches were considered byBraschler and Ferro (2007) to enable multilingual informa-tion retrieval in The European Library (TEL). Moreover,Francis (2008) proposed a set of recommendationsregarding the use of technology for enabling multilingualinformation retrieval in Digital Libraries. Singh (2008)emphasized the language requirements needed to build aMultilingual Digital Library. In addition, Mizera-Pietraszko(2009) focused on comparing bilingual to monolingual har-vesting techniques in the context of multilingualrepositories.

Bosca and Dini (2010) proposed a new approach of usinglog data to infer new translations to enrich existing dic-tionaries based on the assumption that the same query isplaced across languages in a Multilingual Digital Library.Stiller, Gade, and Petras (2010) used log data to analyze thequeries content and language whereas Stiller (2011)focused on useresystem interaction as a means toimprove multilingual information retrieval. In the samecontext, Budzise-Weaver, Chen, and Mitchell (2012) studiedthe key features of such technology and suggested strate-gies for developing and managing Multilingual Digital Li-braries. Finally, Zeng (2012) used freely available digitalresources to automatically construct multilingual thesauri.

Design, implementation and infrastructures of MDLsSeventeen papers of the relevant literature focused onthe design and implementation of MDLs as well as oncreating either infrastructures, tools, techniques or evenPlease cite this article in press as: Vassilakaki, E., & Garoufallou, E., Mand user-centered studies, information retrieval and user behavior,dx.doi.org/10.1016/j.iilr.2013.07.002methodologies and interfaces to support multilingual in-formation retrieval in Digital Libraries. In terms of design,Zhang, Oard, Howarth, and Witten (2005) presented theoutcomes of a session focused on the design of MultilingualDigital Libraries. In addition, Ruecker et al. (2011) andShiri, Ruecker, Fiorentino, Stafford, and Bouchard (2010)presented their designs of a dynamic visual user interfacefor the UNESCO digital portal and libraries. Finally, Yi (2012)focused on building an ontology-driven medical informationretrieval system of the Centers for Disease Control andPreventions medical records.

CACAO (2012) and Oard (1997) focused on infrastructuredevelopment. In particular, Oard (1997) provided detailsregarding the development of a network infrastructure andemphasized the linguistic barriers which were yet to beaddressed whereas CACAO (2012) proposed the combina-tion of natural language processing techniques with currentinformation retrieval systems so as to provide an infra-structure for MDLs.

In addition, Afifi (2000), Andreoni et al. (1999), Fox(2000) and Singh (2008) presented the MDLs which theyhad established. In particular, Andreoni et al. (1999)described their efforts in establishing a European DigitalLibrary for gray literature. Afifi (2000) created a Multilin-gual Digital Library about film in Asia. Fox (2000) estab-lished a multilingual, federated Digital Library forelectronic theses and dissertations aimed for universitystudents whereas Singh (2008) developed a Digital Libraryin Gurmukhi, Shahmukhi, Roman and Devnagri languages.

In terms of techniques and tools developed, Kapidakis,Mavroidis, and Tsalapata (1999) and Pavlov, Nisheva-Pavlova, Lliev, Rousseva, and Apostolova (2010) presentedthe tools they developed for MDLs. In particular, Kapidakiset al. (1999) described an extension of DIENST, a search toolcreated at Cornell University, so as to support multilingualinformation retrieval. On the other hand, Pavlov et al.(2010) presented their efforts to develop software toolsfor multilingual academic Digital Libraries. Finally,Bhardwaj (2010) presented the script-independent andcontent-based retrieval techniques for searching for docu-ment images across languages.

In terms of methodologies and approaches developed,Winiwarter (2000) described the way a machine translationenvironment was developed to enable the automatictranslation of Japanese documents to German whereasShiri, Ruecker, Rosello, Bouchard, and Mehta (2007)focused on the development of a visual user interfacewhich employed a thesaurus to enable multilingual infor-mation retrieval. XEROX (2008, 2010) focused on developingapproaches for enabling retrieval across languages in DigitalLibraries as well as Online Public Access Catalogues(OPACs). Finally, Pavlov et al. (2010) developed a meth-odology of how to build a MDL for universities whereasPROMISE (2011) focused on creating an automated evalua-tion methodology making use of log files.

Implementation problemsFour papers addressed the problems of implementing Digi-tal Libraries. Biagioni et al. (1998) outlined the way aprototype infrastructure for networked access was imple-mented to enable access to ERCIM members technicaldocuments. Zhang et al. (2005) outlined the issues raised inultilingual Digital Libraries: A review of issues in system-centeredThe International Information & Library Review (2013), http://


+ MODELa session regarding the implementation of MDLs. Agosti,Braschler, Ferro, Peters, and Siebinga (2007) describedthe process of implementing multilingual informationretrieval in The European Library (TEL). Finally, Yi (2012)discussed the way a medical ontology was implementedto enable multilingual information retrieval of medicalrecords.

Multilingual information retrieval in Digital LibrariesThe majority of the papers (38 in total) in this sectionpresented various efforts in enabling multilingual informa-tion retrieval in Digital Libraries. In particular, Dartois et al.(1997) and Europeana (2009) described the way multilingualcollections were developed as well as their key character-istics. Moreover, Pavani (2001) described the case of aMultilingual Digital Library in the Maxwell System in PUC-Rio with an emphasis on its key characteristics and itscurrent status. In the same context, Xie and Wu (2009)described the way libraries and librarians could be trainedto provide multilingual information services to users.Furthermore, Larson, Gey, and Chen (2002) provided de-tails about the customization of bilingual dictionaries andmultilingual query mappings of a specific language to manydiverse languages. Xu (2003) focused on the strategiesdeveloped to enable cross and multilingual informationretrieval and access. Bosca and Dini (2006) described thealgorithm they developed to enrich translations fordictionary-based multilingual information retrieval.

In terms of methods and architectures presented,Brisaboa, Penabad, Places, and Rodriguez (2001; Brisaboa,Parama, Penabad, Places, & Rodrguez, 2002) focused onan architecture that was based on ontologies to combinethe multilingual collections of pre-existing Digital Libraries.In addition, Bamman, Babeu, and Crane (2010) outlinedtheir method for automatically providing structural infor-mation across translations; Jones, Fuller, Newman, andZhang (2011) presented a hybrid translation method toenable effective query translation for cultural collections.In particular, they proposed the use of domain-specificphrase dictionaries automatically mined from Wikipedia toenhance the translation mechanism of a standard machinetranslation system.

A few studies described efforts concerning systemsemployed to support multilingual information retrieval inDigital Libraries. Bainbridge, Edgar, Mcpherson, and Witten(2003), Nichols, Witten, Keegan, Bainbridge, and Dewsnip(2005), Nichols, Rose, Bainbridge, and Witten (2010),Shivaram (2002), Witten (1997) and Witten and Bainbridge(2007) described the Greenstone software for developingand supporting Multilingual Digital Libraries and empha-sized its characteristics, features and strengths. Yang et al.(2000) outlined the way the system interface functioned tosupport the indexing of Chinese web pages. In addition,Roussey, Calabretto, and Pinon (2001) presented a systemthat supported multilingual information retrieval makinguse of a knowledge representation model whereas Rauber,Dittenbach, and Merkl (2001) outlined the way the 620/LEDigital Library system was used to support the multilingualcollection of the Russian information agency Novosti.Finally, Xu (2003) presented MIETTA a web-based system inthe tourism domain. An emphasis was given to the design,search engine and natural languages techniques employed.Please cite this article in press as: Vassilakaki, E., & Garoufallou, E., Mand user-centered studies, information retrieval and user behavior,dx.doi.org/10.1016/j.iilr.2013.07.002Moreover, Chen and Ruiz (2009) described the interac-tive framework so as to support cross-language informationaccess and retrieval. Ogrodniczuk and Karagiozov (2011)outlined the ATLAS platform that integrated tools for agroup of European languages. In addition, Ghorab et al.(2011) presented a framework for adaptive multilingualinformation retrieval whereas Stankovic et al. (2012) pre-sented Biblisa a tool for queries enhancement.

In terms of research programs and projects, Klavans(1998) provided details of the outcomes of the firstmeeting of a working group on multilingual informationaccess for Digital Libraries. In addition, Chen (2001)described Chinese Memory Net (CMNet) and focused onthe initial research and research outcomes of the project.Klavans (2001) provided an overview of both ColumbiaUniversitys Digital Library research program as well asdetails of three other projects of multilingual informationaccess. Building on this, Sugimoto (2001) outlined a seriesof projects which were all conducted at University of Li-brary and Information Science (ULIS) and aimed to assistusers while searching for information in multiple languages.Moreover, Chen (2007) presented the Global Memory Net(GMNet) a project for the development of an internationalDigital Library. Finally, Monroy, Furuta, and Castro (2007)described an approach for handling multilingual collec-tions of manuscripts whereas CACAO (2012) provided anapproach for enabling multilingual information retrievaland access in Digital Libraries and catalogs.

In terms of techniques presented, Sheridan, Braschler,and Schauble (1997) outlined the application of a tech-nique which employed thesauri to enable multilingual in-formation retrieval in the domain of Swiss law. Vikas (2005)outlined language policies and computing initiativesregarding multilingualism in India. Gey, Kando, Lin, andPeters (2006) outlined the practical issues that emergedfrom scalability and practical application of multilingualinformation retrieval in Digital Libraries and web portals. Inaddition, Francis (2008) described a set of recommenda-tions for enabling information retrieval across languages inDigital Libraries of developing countries. Furthermore,Tripathi (2008) presented the inherent problems of Indictext processing and the difficulties in building Digital Li-braries for Indian languages. Finally, Daranyi, Wittek, andDobreva (2010) proposed the use of a 5M model as a bestpractice in the context of MDLs.

Evaluation and testingEvaluation and testing of multilingual information retrievalin Digital Libraries was the focus of five studies: Biagioniet al. (1998) tested a prototype infrastructure for net-worked access. As a result they provided multilingual ac-cess and browsing functionalities with the view to adoptUNICODE so as to fully internationalize the system; Evansand Klavans (2003) presented results from an initial evalu-ation of SimFinderML system against monolingual SimFinderand found that SimFinderML had a better precision butworse recall; Chen and Bao (2009) reported an evaluationstudy of Googles cross-language search feature. Theyargued that this feature provided quality query translationbut further development was needed for the automatictranslation of search results; Stiller (2010) focused on theevaluation of user input as a mean to harness useresystemultilingual Digital Libraries: A review of issues in system-centeredThe International Information & Library Review (2013), http://

support retrieval across languages becomes even more ur-


+ MODELinteraction. He stressed the need to involve users in theevaluation process of the multilingual information retrievalsystems; and Mahajan and Singh (2012) evaluated onlineresources such as dictionaries, encyclopedia, digital li-braries in the languages of Hindi, Punjabi and Sanskrit interms of meeting users information needs and provided arelevant list of resources.

Approaches, mechanisms and modelsThe literature of MDLs proposed new approaches, mecha-nisms and even models for enhancing multilingual infor-mation retrieval. The notion of adopting a specific model,was identified in Lee, Sugimoto, Nagamori, Sakaguchi, andTabata (2003) where they suggested the use of a collabo-rative model for the purposes of a multilingual subjectgateway. In particular, Wang, Lu et al. (2004) suggested theuse of a query-translation technique so as to better supportmultilingual information retrieval in Digital Libraries.Building on the query-translation technique, Mizera-Pietraszko (2009) proposed a more user-oriented approachto enable multilingual information retrieval. In the samecontext, Clinchant and Renders (2009) suggested that thedictionary adaption mechanism should be extended toinclude various languages and that a multi-language modelshould be adopted. Finally, the papers of Budzise-Weaveret al. (2012) and Chen and Bao (2009) proposed a seriesof strategies with the view to developing a sustainable MDL.

User-centered studies

The literature on MDLs was divided in system-centered anduser-centered studies. Although, a significant amount ofstudies were published concerning the various technicalaspects of Multilingual Digital Libraries, only a small num-ber of studies focused on user interaction.

User expectations and needsFour studies tried to shed light on user expectations andneeds when using a MDL. Wu (2010) and Wu, Luo, and He(2010) used a mixture of qualitative and quantitativemethods aiming to explore users expectations whensearching across languages in Chinese academic libraries. Inaddition, Gade (2011) employed Europeana ClickstreamLogger to look into user behavior by mining log data with theview to propose a design for multilingual informationretrieval systems. Finally, Wu, He, and Luo (2012) employedquestionnaires to investigate academic users expectationsand needs when searching in Multilingual Digital Libraries.

User behaviorFour studies focused on users information seeking behaviorwhen searching for information across languages. Bilal andBachir (2007) employed a mixture of methods to exploreArabic native speaking childrens search strategies in theInternational Childrens Digital Library. Moreover, Ghorab,Leveling, Zhou, Jones, and Wade (2010) examined logfiles to identify different information seeking behaviorbased on different linguistic or cultural backgrounds and tocreate communities based on users common search pat-terns. Log files were also used by Takaku et al. (2010) toexplore user behavior. They treated user behavior as aprocess which consisted of sequences of actions andPlease cite this article in press as: Vassilakaki, E., & Garoufallou, E., Mand user-centered studies, information retrieval and user behavior,dx.doi.org/10.1016/j.iilr.2013.07.002gent given the increasing interaction between differentcultures. The literature identified mainly focuses on twoaspects, the System and the Users. As such, the rele-vant papers were divided in two subject themes:

a) the system-centered studies, which aimed to investi-gate and provide possible solutions to the technical as-pects for enabling multilingual information retrieval andb) the user-centered studies that set out to exploreusers behavior and expectations when searching inmultilingual environments with the view to designingefficient and effective interfaces.

Further analysis of the papers, led to the emergence ofsub-themes in various research areas. In terms of system-centered aspect, six main issues of research were identi-fied. In particular, challenges raised for enabling multilin-gual information retrieval in Digital Libraries wereexplored. In this context, a series of features (Budzise-Weaver et al., 2012), approaches (Bosca & Dini, 2010),models (Cheng et al., 1999) and techniques (Mizera-Pietraszko, 2009) were proposed as ways to meet thesechallenges. All these attempts sought to provide a means tooffer efficient and effective information retrieval in Multi-lingual Digital Libraries.

In addition, efforts focused on designing (Afifi, 2000;Andreoni et al., 1999; Fox, 2000; Singh, 2008; Zhang et al.,durations. Finally, Petrelli and Clough (2012) studied thequeries that users generated while searching a historicimage archive collection in multiple languages.

User knowledge of languagesOne study by Clough and Eleta (2010) explored the waysknowledge of foreign languages could affect users infor-mation seeking behavior. It focused on the effects of lan-guage skills and field of knowledge when users sought forinformation across multiple languages.

Interface usageFinally, four studies explored users behavior with the viewto monitor and evaluate Multilingual Digital Librariesinterface usage. In particular, Stafford, Shiri, Ruecker,Bouchard, and Mehta (2008) and Shiri, ruecker, Doll,Bouchard, & Fiorentino (2011) employed qualitativemethods and asked 15 users to search across languages onSearchling using a bilingual thesaurus. Building on thisresearch, Shiri, Ruecker, Bouchard et al. (2011) crossevaluated two interfaces for Digital Libraries T-Saurus andSearchling again by employing a mixture of methods andlooking into user behavior. Finally, Sastry, Manjunath, andReddy (2011) adopted a variety of both qualitative andquantitative methods in an effort to study users informa-tion seeking behavior and provide guidelines for designinginterfaces for university Multilingual Digital Libraries.

Discussion & future work

The literature concerning Multilingual Digital Libraries, asalready outlined, is vast and grows rapidly (see Annex A,Table 1). This comes as no surprise, since the need toultilingual Digital Libraries: A review of issues in system-centeredThe International Information & Library Review (2013), http://

2005), creating and implementing infrastructures (CACAO,2012; Oard, 1997), techniques (Bhardwaj, 2010), tools(Kapidakis et al., 1999; Pavlov et al., 2010), methodologies(Winiwarter, 2000) and interfaces (Ruecker et al., 2011; Shiriet al., 2010) to support Multilingual Digital Libraries. Theseefforts undertaken by a small number of researchers orresearch groups apparently working in isolation and there-fore resulting in the development of numerous different toolsand techniques.

Furthermore, examples of how Multilingual Digital Li-braries were implemented (Agosti, Braschler et al., 2007;Zhang et al., 2005) were reported to illustrate the issuesand main problems behind such a difficult task. In terms ofdescribing multilingual information retrieval in Digital Li-braries (Larson et al., 2002; Pavani, 2001), differentmethods and architectures were presented mainly on howtranslations could be generated and provided to users(Jones et al., 2011). In this context, Greenstone was themain system employed and presented to support multi-lingual information retrieval (Bainbridge et al., 2003;Shivaram, 2002; Witten, 1997); different frameworkswere illustrated (Chen & Ruiz, 2009; Stankovic et al.,2012); and different projects and programs were outlined(Klavans, 1998; Monroy et al., 2007). All these effortsdemonstrate the diversity of the research undertaken, aswell as that each group acted independently from therest.

Finally, the issues emerging from applying multilingualinformation retrieval were raised through the testing ofvarious systems and techniques (Evans & Klavans, 2003;

Stiller, 2010) and further aproaches (Mizera-Pietraszko,2009), mechanisms (Wang, Lu et al., 2004) and models(Lee et al., 2003) were proposed for enhancing multilin-gual information retrieval and access.

As a result, the need to provide common models,techniques, approaches, methodologies, and developcommon infrastructures for building and implementingMultilingual Digital Libraries becomes more urgent thanever. It is these protocols and common rules which willenable interoperability and communication betweendifferent systems and openness to the systems to differentresearch groups.

Compared to system-centered studies, relatively fewstudies have focused on the user and aimed to exploreusers behavior and expectations when interacting withMultilingual Digital Libraries. The majority of thesestudies employed mainly log analysis (Gade, 2011; Ghorabet al., 2010), or a qualitative method which tends togeneralize findings whereas a mixture of methods wereemployed by others (Wu, 2010; Wu et al., 2010). The maingoal of these studies was to shed light on users infor-mation seeking behavior while searching in a multilingualenvironment and provide guidelines in designing in-terfaces. Further research is needed to reach to sometangible and usable findings which will contribute to thedesign of efficient, effective and user-friendly interfaces.

Annex A. Tables

Table 1 Number of identified relevant papers per year.

et

Kapang, 20

Shi00200al.etir,2020ch

caavt aal.et ard e

; Hkov


+ MODELYear No. Papers Papers

1997 6 Dartois et al., 1997; Oard, 1997; Sheridan1998 3 Biagioni et al., 1998; Klavans, 19981999 4 Andreoni et al., 1999; Cheng et al., 1999;2000 4 Afifi, 2000; Fox, 2000; Winiwarter, 2000; Y2001 7 Brisaboa et al., 2001; Chen, 2001; Klavans

Rauber et al., 2001; Roussey et al., 2001;Sugimoto, 2001

2002 3 Brisaboa et al., 2002; Larson et al., 2002;2003 5 Bainbridge et al., 2003; Evans & Klavans, 22004 5 Wang, Lu et al., 2004; Wang, Teng et al.,2005 3 Nichols et al., 2005; Vikas, 2005; Zhang et2006 4 Bosca & Dini, 2006; Gey et al., 2006; Wang2007 10 Agosti, Braschler et al., 2007; Bilal & Bach

2007; Mizera-Pietraszko, 2007; Shiri et al.,2008 5 Francis, 2008; Singh, 2008; Stafford et al.,2009 7 Chen & Bao, 2009; Chen & Ruiz, 2009; Clin

2009; Xie & Wu, 20092010 16 Bamman et al., 2010; Bhardwaj, 2010; Bos

Ghorab et al., 2010; Nichols et al., 2010; P2010; Takaku et al., 2010; Wu, 2010; Wu e

2011 14 Gade, 2011; Ghorab et al., 2011; Jones etOgrodniczuk & Karagiozov, 2011; RueckerSastry et al., 2011; Shiri, Ruecker, BouchaDoll et al., 2011; Stiller, 2011

2012 10 Budzise-Weaver et al., 2012; CACAO, 2012Peters, 2012; Petrelli & Clough, 2012; StanYi, 2012; Zeng, 2012Please cite this article in press as: Vassilakaki, E., & Garoufallou, E., Mand user-centered studies, information retrieval and user behavior,dx.doi.org/10.1016/j.iilr.2013.07.002al., 1997; Witten, 1997

idakis et al., 1999et al., 200001; Pavani, 2001;

varam, 20023; Lee et al., 2003; Xu, 20034, 2005al., 20062007; Braschler & Ferro, 2007; Chen, 2007; Monroy et al.,07; Witten & Bainbridge, 200708; Tripathi, 2008; XEROX, 2008ant & Renders, 2009; Europeana, 2009; Mizera-Pietraszko,

& Dini, 2010; Clough & Eleta, 2010; Daranyi et al., 2010;lov et al., 2010; Shiri et al., 2010; Stiller, 2010; Stiller et al.,l., 2010; XEROX, 2010, 2011;l., 2011;t al., 2011; Shiri, Ruecker,

e, 2012; Mahajan & Singh, 2012;ic et al., 2012; Wu et al., 2012;ultilingual Digital Libraries: A review of issues in system-centeredThe International Information & Library Review (2013), http://

In Table 1, all identified papers relevant to MultilingualDigital Libraries were assigned to each year of publication.As it could be concluded, there is a steady increase in theinterest of research surrounding the various aspects ofMDLs.

In Table 2, a methodological analysis of the papers assignstoeachyearofpublicationwasconducted. Inparticular, itwasfound that Case study is the predominant method employedwhen exploring issues in MDLs. It was not until 2010 and on-wards that a mixture of different methods was adopted.

Table 2 Methods used in the identified papers per year.

Year Methods

1997 Case study (4)1998 Case study (2)1999 Case study (3)2000 Case study (4)2001 Case study (7)2002 Case study (3)2003 Case study (4)2004 Case study (2)2005 Session overview, Case study (2)2006 Case study (2), Log analysis2007 Case study (7), Interview2008 Interview, Thinkaloud Protocol, Case study (4)2009 Case study (5), Usability Testing2010 Log analysis (4), Case study (8), Interview (2), Questionnaire (3)2011 Case study (5), Log analysis (4), Usability Test (2), Workshop overview, Questionnaire (2), Observation (2),

Interview (2), Thinkaloud protocol (2), Ethnography, Focus Group2012 Log analysis, Questionnaire (2), Observation, Text Mining, Case study (5), Special Issue overview

Table 3 Identified papers type of publication.

Kind of paper No.papers

Papers Specific source

Proceedingspaper

38 Agosti, Braschler et al., 2007; Andreoniet al., 1999; Bamman et al., 2010;Bosca & Dini, 2006; Bosca & Dini, 2010;Brisaboa et al., 2001; Brisaboa et al., 2002;Chen, 2007; Chen & Bao, 2009; Chen &Ruiz, 2009; Clinchant & Renders, 2009;Daranyi et al., 2010; Evans & Klavans, 2003;Ghorab et al., 2010; Ghorab et al., 2011;Klavans, 2001; Larson et al., 2002; Lee et al.,2003; Nichols et al., 2010; Mizera-Pietraszko,2009; Rauber et al., 2001; Roussey et al.,2001; Shiri et al., 2007; Shiri et al., 2010;Shiri, Ruecker, Bouchard et al., 2011;Stafford et al., 2008; Stankovic et al.,2012; Stiller, 2010; Stiller et al., 2010;Sugimoto, 2001; Takaku et al., 2010; Vikas,2005; Wang, Teng et al., 2004; Winiwarter,2000; Wu, 2010; Wu et al., 2010; Xie & Wu,2009; Zhang et al., 2005

Annual Conference of the Canadian Association forInformation Science (2), Applications of DigitalInformation and Web Technologies, 2009. ICADIWT09.Second International Conference on the, CLEF (5),ECDL (2), ECIR 2011, Electronic publishing: 2001 inthe Digital Publishing Odyssey, EurAsia-ICT 2002:information and communication technology, IFLAGovernment Libraries Section, Library Services toMulticultural Populations Section, JCDL (6),Management and Service Science (MASS), 2010International Conference on, Proceeding of the 33rdinternational ACM SIGIR conference on Researchand development in information retrieval e SIGIR10,Proceedings 2000 Kyoto International Conferenceon Digital Libraries: Research and Practice (2),Proceedings of NTCIR-6 Workshop Meeting,Proceedings of the 2003 international conferenceon Dublin Core and metadata applications: supportingcommunities of discourse and practicemetadataresearch \& applications (DCMI), Proceedings of theAmerican Society for Information Science andTechnology (2), Proceedings of the Eighth InternationalConference on Language Resources and Evaluation(LREC), Proceedings of the Fourth International


+ MODELPlease cite this article in press as: Vassilakaki, E., & Garoufallou, E., Mand user-centered studies, information retrieval and user behavior,dx.doi.org/10.1016/j.iilr.2013.07.002Conference on Asian Digital Library, Proceedings ofthe International Conference on Semantic Web andDigital Libraries, Proceedings of the RCDL 2001

(continued on next page)ultilingual Digital Libraries: A review of issues in system-centeredThe International Information & Library Review (2013), http://

(continued )

Kind of paper No.papers

Papers Specific source

"Digital Libraries: Advanced Methods and Technologies,Digital Collections", Second International Conferenceon Education Technology and Training, SIGIR Workshopon Information Access in a Multilingual World, TheFifth Annual iConference, Champaign, Third All-RussianConference Digital Libraries: Advanced Methods andTechnologies, Digital Collections Petrozavodsk,RCDI2001, TPDL (2), UNESCO

Journalpaper

27 Bilal & Bachir, 2007; Budzise-Weaver et al.,2012; Cheng et al., 1999; Clough & Eleta,2010; Francis, 2008; Gade, 2011; Kapidakiset al., 1999; Klavans, 1998; Nichols et al.,2005; Mahajan & Singh, 2012; Oard, 1997;Ogrodniczuk & Karagiozov, 2011; Pavani,

ugh.,

en,

Bulletin of IEEE Technical Committee on DigitalLibraries (2), CALIBER, Ciencia da Informacao, D-LibMagazine (2), Digital Libraries, FORTH e ICS TR-248,IEEE-CS Technical Committee on Digital LibrariesBulletin, Information Development, InformationProcessing & Management, International Journal of


+ MODEL2001; Pavlov et al., 2010; Petrelli & Clo2012; Ruecker et al., 2011; Sastry et al2011; Shiri, Ruecker, Doll et al., 2011;Stiller, 2011; Tripathi, 2008; Wang,Lu et al., 2004; Wang et al., 2006; WittIn Table 3, the relevant papers were assigned tocategories based on their type of publication. A varietyof different types of publications were identified, aswell as a variety of the sources of publication.

1997; Wu et al., 2012; Yang et al., 2000;Yi, 2012; Zeng, 2012

Booksection

8 Afifi, 2000; Bainbridge et al., 2003;Braschler & Ferro, 2007; Dartois et al.,1997; Jones et al., 2011; Monroy et al.,2007; Sheridan et al., 1997; Xu, 2003

Report 9 Biagioni et al., 1998; CACAO, 2012;Chen, 2001; Europeana, 2009; Fox,2000; Gey et al., 2006; Shivaram,2002; XEROX, 2008; XEROX, 2010

Thesis 2 Bhardwaj, 2010; Singh, 2008Wiki entry 1 Witten & Bainbridge, 2007

Table 4 Emerged themes & assigned papers.

A/A Themes

1 System-centeredstudies

1.1.Exploration andchallenges

1.2. Implementationand infrastructuresof MLDLs

Please cite this article in press as: Vassilakaki, E., & Garoufallou, E., Mand user-centered studies, information retrieval and user behavior,dx.doi.org/10.1016/j.iilr.2013.07.002, Computer Applications, International Journal ofDigital Libraries, International Journal of DigitalLibraries Systems, Journal of the American Societyfor Information Science (3), Journal of the ChicagoColloquium on Digital Humanties and ComputerNonetheless, most papers were published in the contextof conference proceedings and journals. Again, no pref-erence to a specific conference or specific journal wasidentified.

Science, Library Philosophy and Practice, New Reviewof Hypermedia and Multimedia, Pregled Nacionalnogcentra za digitalizaciju, Procesamiento del LenguajeNatural, The Electronic Library (7), TUGboatResearch and Advanced Technology for DigitalLibraries (7), Knowledge-based information retrievaland filtering from the Web

No. papers Papers

13 Braschler & Ferro, 2007; Bosca & Dini,2010; Budzise-Weaver et al., 2012;Cheng et al., 1999; Francis, 2008;Mizera-Pietraszko, 2009; Stiller, 2011;Stiller, Gade & Petras, 2010; Singh, 2008;Wang, Lu et al., 2004; Wang, Teng et al.,2004; Wang et al., 2006; Yang et al.,2000; Zeng, 2012

17 Afifi, 2000; Andreoni et al., 1999;Bhardwaj, 2010; CACAO, 2012; Fox,2000; Kapidakis et al., 1999;Ruecker et al., 2011; Oard, 1997;Pavlov et al., 2010; Shiri et al.,2007; Shiri et al., 2010; Singh, 2008;

ultilingual Digital Libraries: A review of issues in system-centeredThe International Information & Library Review (2013), http://

(continued )

A/A Themes No. papers Papers

Winiwarter, 2000; XEROX, 2008; XEROX,2010; Yi, 2012; Zhang et al., 2005

1.3. Implementation 4 Agosti et al., 2007; Biagioni et al.,


+ MODELproblems1.4.Multilingualinformationretrieval in DigitalLibrariesIn Table 4, the emerged themes based on the papersexpressed aims and objectives were outlined. In particular,two major themes were identified that of system and usercentered studies. These two themes were further catego-rized into sub-themes again based on the detailed expres-sion of aims. The system centered studies covered a varietyof different aspects of Multilingual Digital Libraries from

1.5. Evaluationand testing

1.6. Approaches,mechanismsand models

2 User-centeredstudies

2.1. User expectationsand needs2.2. User behavior

2.3. User knowledge oflanguages2.4. Interface usage

Table 5 Methods employed in the assigned papers of the emerged

A/A Themes No. Papers Paper


1.1.Exploration andchallenges

13 Bosca2007;

Please cite this article in press as: Vassilakaki, E., & Garoufallou, E., Mand user-centered studies, information retrieval and user behavior,dx.doi.org/10.1016/j.iilr.2013.07.0021998; Yi, 2012; Zhang et al., 200538 Bamman et al., 2010; Bainbridge et al.,

2003; Bosca & Dini, 2006; Brisaboa et al.,2001; Brisaboa et al., 2002; CACAO, 2012;Chen, 2001; Chen, 2007; Chen & Ruiz,2009; Daranyi et al., 2010; Dartois et al.,1997; Europeana, 2009; Francis, 2008;Ghorab et al., 2011; Gey et al., 2006;design and implementation to evaluation. There was asignificant number of papers published focusing on simplypresentation of MDL. In terms of user centered studies, therelevant literature identified was fewer but at the sametime diverse as it covered among others user expectations,requirements, knowledge of languages, and interface usagewith a particular focus on users behaviors.

Jones et al., 2011; Klavans, 1998;Klavans, 2001; Larson et al., 2002;Monroy et al., 2007; Nichols et al.,2005; Nichols et al., 2010;Ogrodniczuk & Karagiozov, 2011;Pavani, 2001; Rauber et al., 2001;Roussey et al., 2001; Sheridan et al.,1997; Shivaram, 2002; Stankovic et al.,2012; Sugimoto, 2001; Tripathi, 2008;Vikas, 2005; Witten, 1997; Witten& Bainbridge, 2007; Xie & Wu, 2009;Xu, 2003; Yang et al., 2000

5 Biagioni et al., 1998; Chen & Bao,2009; Evans & Klavans, 2003;Mahajan & Singh, 2012; Stiller, 2010

6 Budzise-Weaver et al., 2012; Chen& Bao, 2009; Clinchant & Renders,2009; Mizera-Pietraszko, 2009;Lee et al., 2003; Wang, Lu & Chien,2004

4 Gade, 2011; Wu, 2010; Wu et al.,2010; Wu et al., 2012

4 Bilal & Bachir, 2007; Ghorab et al.,2010; Petrelli & Clough, 2012;Takaku et al., 2010

1 Clough & Eleta, 2010

4 Sastry et al., 2011; Shiri,Ruecker, Bouchard et al.,2011; Shiri, Ruecker,Doll et al., 2011; Staffordet al., 2008

themes.

s Methods

& Dini, 2010; Braschler & Ferro,Budzise-Weaver et al., 2012;

Log analysis (3),Case study (8),

(continued on next page)


(continued )

A/A Themes No. Papers Papers Methods

Cheng et al., 1999; Francis, 2008;Mizera-Pietraszko, 2009; Singh, 2008; Stiller,2011; Stiller et al., 2010; Wang, Lu et al.,2004; Wang, Teng et al., 2004; Wang et al.,2006; Yang et al., 2000; Zeng, 2012

Usability criteria,Text mining

1.2. Implementation andinfrastructures of MLDLs

17 Afifi, 2000; Andreoni et al., 1999; Bhardwaj,2010; CACAO, 2012; Fox, 2000;Kapidakis et al., 1999; Oard, 1997; Pavlovet al., 2010; Ruecker et al., 2011; Shiri et al.,2007; Shiri et al., 2010; Singh, 2008;Winiwarter, 2000; XEROX, 2008; XEROX, 2010;Yi, 2012; Zhang et al., 2005

Case study (17)

1.3. Implementationproblems

4 Agosti et al., 2007; Biagioni et al., 1998;Yi, 2012; Zhang et al., 2005

Session overview

1.4. Multilingualinformationretrieval in DigitalLibraries

38 Bamman et al., 2010; Bainbridge et al.,2003; Bosca & Dini, 2006; Brisaboa et al.,2001; Brisaboa et al., 2002; CACAO, 2012;Chen, 2001; Chen, 2007; Chen & Ruiz, 2009;Daranyi et al., 2010; Dartois et al., 1997;Europeana, 2009; Francis, 2008; Gey et al.,2006; Ghorab et al., 2011; Jones et al., 2011;Klavans, 1998; Klavans, 2001; Larson et al.,2002; Monroy et al., 2007; Nichols et al.,2005; Nichols et al., 2010; Ogrodniczuk &Karagiozov, 2011; Pavani, 2001; Rauber et al.,2001; Roussey et al., 2001; Sheridan et al.,1997; Shivaram, 2002; Stankovic et al.,2012Sugimoto, 2001; Tripathi, 2008; Vikas,2005; Witten, 1997; Witten & Bainbridge,2007; Xie & Wu, 2009; Xu, 2003; Yanget al., 2000

Log analysis (2),Case study (36),Workshopoverview (2)

1.5. Evaluation andtesting

5 Biagioni et al., 1998; Chen & Bao, 2009;Evans & Klavans, 2003; Mahajan & Singh,2012; Stiller, 2010

Case study (5)

1.6. Approaches,mechanismsand models

6 Budzise-Weaver et al., 2012; Chen & Bao,2009; Clinchant & Renders, 2009; Lee et al.,2003; Mizera-Pietraszko, 2009; Wang,Lu et al., 2004

Case study (5),Usability criteria


2.1. Userexpectations andneeds

4 Gade, 2011; Wu, 2010; Wu et al., 2010;Wu et al., 2012

Log analysis,Interview (2),Questionnaire (3)

2.2. User behavior 4 Bilal & Bachir, 2007; Ghorab et al., 2010;Petrelli & Clough, 2012; Takaku et al.,2010

Log analysis (3),Questionnaire,Observation,Case study,Interview

2.3. User knowledgeof languages

1 Clough & Eleta, 2010 Questionnaire

2.4. Interface usage 4 Sastry et al., 2011; Shiri, Ruecker,Bouchard et al., 2011; Shiri, Ruecker,Doll et al., 2011; Stafford et al., 2008

Interview (3),Thinkaloudprotocol (3),Questionnaire (2),Observation (2),Usability testing,Ethnographicstudies, Focusgroup, Log analysis


+ MODEL

Please cite this article in press as: Vassilakaki, E., & Garoufallou, E., Multilingual Digital Libraries: A review of issues in system-centeredand user-centered studies, information retrieval and user behavior, The International Information & Library Review (2013), http://dx.doi.org/10.1016/j.iilr.2013.07.002

In Table 5, for each identified category and sub-categories, the methods employed by the assigned papersare illustrated. The quantitative methods (Log Analysis and

Questionnaire) are the most commonly used whereasqualitative methods (interviews, observation) are morescarcely employed.

Table 6 Papers type of publication & specific source.

A/A Themes No.papers

Papers Kind of papers Specific source


1.1.Explorationand challenges

13 Bosca & Dini, 2010;Braschler & Ferro, 2007;Budzise-Weaver et al.,2012; Cheng et al., 1999;Francis, 2008; Mizera-Pietraszko, 2009; Singh,2008; Stiller, 2011; Stilleret al., 2010; Wang, Luet al., 2004; Wang, Tenget al., 2004; Wang et al.,2006; Yang et al., 2000;Zeng, 2012

Proceedings paper(4), Journal paper(7), Book section,Master thesis

Applications of DigitalInformation and WebTechnologies, 2009.ICADIWT09. SecondInternational Conference onthe, Bulletin of IEEE TechnicalCommittee on DigitalLibraries, CLEF (2), DigitalLibraries: R&D, InformationDevelopment, Journal of theAmerican Society forInformation Science (3),Proceedings of the 2004 JointACM/IEEE Conference onDigital Libraries (JCDL04), TheElectronic Library (2)

1.2.Implementationandinfrastructuresof MLDLs

17 Afifi, 2000; Andreoniet al., 1999; Bhardwaj,2010; CACAO, 2012; Fox,2000; Kapidakis et al.,1999; Oard, 1997; Pavlovet al., 2010; Rueckeret al., 2011; Shiri et al.,2007; Shiri et al., 2010;Singh, 2008; Winiwarter,2000; XEROX, 2008;XEROX, 2010; Yi, 2012;Zhang et al., 2005

Proceedings paper(5), Journal paper(3), Project review(2), Report (3),Book section,Master thesis, PhDthesis

Annual Conference of theCanadian Association forInformation Science (2), D-LibMagazine, ECDL99, FORTH-ICSTR-248, Journal of the ChicagoColloquium on DigitalHumanities and ComputerScience, Pregled Nacionalnogcentra za digitalizaciju,Proceedings 2000 KyotoInternational Conference onDigital Libraries: Research andPractice, Proceedings of theAmerican Society forInformation Science andTechnology, Research andAdvanced Technology forDigital Libraries, TheElectronic Library

1.3. Implementationproblems

4 Agosti et al., 2007;Biagioni et al., 1998; Yi,2012; Zhang et al., 2005

Proceedings paper(2), Journal paper,Project report

ECDL, Proceedings of theAmerican Society forInformation Science andTechnology, The ElectronicLibrary

1.4.Multilingualinformationretrieval inDigital Libraries

38 Bamman et al., 2010;Bainbridge et al., 2003;Brisaboa et al., 2001;Brisaboa et al., 2002;Bosca & Dini, 2006;CACAO, 2012; Chen,2001; Chen, 2007; Chen& Ruiz, 2009; Daranyiet al., 2010; Dartoiset al., 1997; Europeana,2009; Francis, 2008; Gey

oranes

Proceedings paper(17), Book section(5), Project report(2), Journal paper(8), Book chapter,Project report (2),Project review,Wiki entry,Workshop report

ACM SIGIR Forum, AdvancedLanguage Technologies forDigital Libraries, CALIBER,Ciencia da Informacao, DigitalLibraries, Digital Libraries forInternational DevelopmentWorkshop (2), Electronicpublishing: 2001 in the DigitalPublishing Odyssey, EurAsia-ICT 2002: information andcommunication technology,


+ MODELet al., 2006; Ghet al., 2011; JoPlease cite this article in press as: Vassilakaki, E., & Garoufallou, E., Mand user-centered studies, information retrieval and user behavior,dx.doi.org/10.1016/j.iilr.2013.07.002bet al.,

IEEE-CS Technical Committeeon Digital Libraries Bulletin,

(continued on next page)ultilingual Digital Libraries: A review of issues in system-centeredThe International Information & Library Review (2013), http://

(continued )


Papers Kind of papers Specific source

2011; Klavans, 1998;Klavans, 2001; Larsonet al., 2002; Monroyet al., 2007; Nicholset al., 2005; Nicholset al., 2010;Ogrodniczuk &Karagiozov, 2011;Pavani, 2001; Rauberet al., 2001; Rousseyet al., 2001; Sheridanet al., 1997; Shivaram,2002; Stankovic et al.,2012; Sugimoto, 2001;Tripathi, 2008; Vikas,2005; Witten, 1997;Witten & Bainbridge,2007; Xie & Wu, 2009;Xu, 2003; Yang et al.,2000

IFLA Government LibrariesSection, Library Services toMulticultural PopulationsSection at the IFLA Meeting inDurban, InformationDevelopment, JCDL, Journal ofthe American Society forInformation Science,Knowledge-based informationretrieval and filtering from theWeb, New Review ofHypermedia and Multimedia,Proceedings 2000 KyotoInternational Conference onDigital Libraries: Research andPractice, Proceedings of theEighth InternationalConference on LanguageResources and Evaluation(LREC), Proceedings of theFourth InternationalConference on Asian DigitalLibrary, Proceedings of theInternational Conference onSemantic Web and DigitalLibraries, Proceedings of theRCDL 2001 "Digital Libraries:Advanced Methods andTechnologies, DigitalCollections", Proceedings ofthe second ACM/IEEE-CS jointconference on Digital librariese JCDL02, Procesamiento delLenguaje Natural, Researchand Advanced Technology forDigital Libraries (4), SecondInternational Conference onEducation Technology andTraining, SIGIR Workshop onInformation Access in aMultilingual World, Third All-Russian Conference DigitalLibraries: Advanced Methodsand Technologies, DigitalCollections Petrozavodsk,RCDI2001, TPDL, UNESCO

1.5. Evaluationand testing

5 Biagioni et al., 1998;Chen & Bao, 2009;Evans & Klavans, 2003;Mahajan & Singh, 2012;Stiller, 2010

Journal paper,Book section,Proceedingspaper (3)

JCDL, Library Philosophy andPractice, Proceeding of the33rd international ACM SIGIRconference on Research anddevelopment in informationretrieval e SIGIR 10,Proceedings of the AmericanSociety for Information Scienceand Technology, Research andAdvanced Technology forDigital Libraries


+ MODEL

Please cite this article in press as: Vassilakaki, E., & Garoufallou, E., Multilingual Digital Libraries: A review of issues in system-centeredand user-centered studies, information retrieval and user behavior, The International Information & Library Review (2013), http://dx.doi.org/10.1016/j.iilr.2013.07.002

etao,de., 2ko,, 20


+ MODEL(continued )


Papers

1.6. Approaches,mechanisms andmodels

6 Budzise-Weaver2012; Chen & BClinchant & Ren2009; Lee et alMizera-PietraszWang, Lu et al.In Table 6, the specific source of where each paperassigned to each category and subcategory was published,is showed. In particular, the most common source of pub-lication is prestigious international conferences (ECDL,CLEF) and their proceedings.

References

Afifi, M. (2000). Asian film connection: developing a scholarlymultilingual digital library e a case study. In J. L. Borbinha, &T. Baker (Eds.), Research and advanced technology for digitallibraries (pp. 481e484). Berlin/Heidelberg: Springer. http://dx.doi.org/10.1007/3-540-45268-0_67.

Agosti, M., Braschler, M., Ferro, N., Peters, C., & Siebinga, S.(2007). Roadmap for multilingual information access in theeuropean library. In L. Kovacs, N. Fuhr, & C. Meghinni (Eds.),


2.1. Userexpectationsand needs

4 Gade, 2011; Wu, 2Wu et al., 2010; Wet al., 2012

2.2. Userbehavior

4 Bilal & Bachir, 200Ghorab et al., 201Petrelli & Clough,Takaku et al., 2010

2.3. User knowledgeof languages

1 Clough & Eleta, 20

2.4. Interface usage 4 Sastry et al., 2011;Shiri, Ruecker,Bouchard et al., 20Shiri, Ruecker,Doll et al., 2011;Stafford et al., 200

Please cite this article in press as: Vassilakaki, E., & Garoufallou, E., Mand user-centered studies, information retrieval and user behavior,dx.doi.org/10.1016/j.iilr.2013.07.002Kind of papers Specific source

al.,2009;rs,003;2009;04

Journal paper (2),Proceedings paper(4)

Applications of DigitalInformation and WebTechnologies, 2009. ICADIWT09. Second InternationalConference on the, CLEF 2008,International Journal of DigitalLibraries, Proceedings of the2003 international conferenceon Dublin Core and metadataapplications: supportingcommunities of discourse andpracticemetadata researchECDL2007 (pp. 136e147). Berlin/Heidelberg: Springer. http://dx.doi.org/10.1007/978-3-540-74851-9_12.

Agosti, M., Crivellari, F., & Di Nunzio, G. M. (2011). Web log anal-ysis: a review of a decade of studies about information acqui-sition, inspection and interpretation of user interaction. DataMining and Knowledge Discovery, 24(3), 663e696. http://dx.doi.org/10.1007/s10618-011-0228-8.

Agosti, M., Di Nunzio, G. M., Ferro, N., & Peters, C. (2007). CLEF:ongoing activities and plans for the future. In Proceedings ofNTCIR-6 workshop meeting, May 15e18, 2007, Tokyo, Japan(pp. 493e504). Retrieved from http://www.mt-archive.info/NTCIR-2007-Agosti.pdf.

Andreoni, A., Baldacci, M. B., Biagioni, S., Carlesi, C.,Castelli, D., Pagano, P., et al. (1999). Developing a Europeantechnical reference digital library. In S. Abiteboul, &A. M. Vercoustre (Eds.), ECDL99 (pp. 343e362). Berlin/-Heidelberg: Springer.

\& applications (DCMI 03).Dublin Core MetadataInitiative, Proceedings of theAmerican Society forInformation Science andTechnology, The ElectronicLibrary

010;u


Bulletin of IEEE TechnicalCommittee on DigitalLibraries, Management andService Science (MASS), 2010International Conference on,The Electronic Library, TheFifth Annual iConference,Champaign, IL

7;0;2012;


CLEF (2), InformationProcessing & Management,The Electronic Library (2)

10 Journal paper International Journal of DigitalLibraries Systems

11;

8


ECDL, International Journal ofComputer Applications, TheElectronic Library, TPDL


Bainbridge, D., Edgar, K. D., Mcpherson, J. R., & Witten, I. H. (Ed.), Global digital library development in the new millen


+ MODEL(2003). Managing change in a digital library system with manyinterface languages. In Research and advanced technology fordigital libraries (pp. 350e361). Berlin/Heidelberg: Springer.Retrieved from http://greenstonesupport.iimk.ac.in/greenstone2009/program-cd/pdf/articles/DB-KE-JMcP-IHW-Managingchange.pdf.

Bamman, D., Babeu, A., & Crane, G. (2010). Multilingual alignmentand projection. In JCDL2010 (pp. 1e10). Retrieved fromhttp://www.perseus.tufts.edu/wamahoney/jcdl27-bamman.pdf.

Bhardwaj, A. (2010). Statistical techniques for efficientindexing and retrieval of document images. Retrieved fromhttp://udini.proquest.com/view/statistical-techniques-for-goid:759928901/.

Biagioni, S., Borbinha, J. L., Ferber, R., Hansen, P., Kapidakis, S.,Kovacs, L., et al. (1998). The ERCIM technical reference digitallibrary. In Research and advanced technology for digital li-braries (pp. 905e906). Berlin/Heidelberg: Springer.

Bilal, D., & Bachir, I. (2007). Childrens interaction with cross-cultural and multilingual digital libraries. II. Informationseeking, success, and affective experience. Information Pro-cessing & Management, 43(1), 65e80. http://dx.doi.org/10.1016/j.ipm.2006.05.008.

Borgman, C. L. (1997). Multi-Media, Multi-Cultural, and Multi-Lingual Digital Libraries Or How Do We Exchange Data In 400Languages? US: D-Lib Magazine. Retrieved from http://dlib.org/dlib/june97/06borgman.html.

Borgman, C. L. (2000). From Gutenberg to the global informationinfrastructure: Access to information in the networked world.US: MIT Press.

Bosca, A., & Dini, L. (2006). The role of logs in improving crosslanguage access in digital libraries. In Proceedings of the in-ternational conference on semantic web and digital libraries,2009 (pp. 1e11). Retrieved from http://research.celi.it/JSPWiki/attach/Publications/icsd2.pdf.

Bosca, A., & Dini, L. (2010). User logs as a means to enrich andrefine translation dictionaries. In C. Peters, et al. (Eds.),CLEF2009 workshop (pp. 544e551). Berlin/Heidelberg:Springer. http://dx.doi.org/10.1007/978-3-642-15754-7_67.

Braschler, M., & Ferro, N. (2007). Adding multilingual informationaccess to the European Library. In C. Thanos, F. Borri, &L. Candela (Eds.), Digital libraries: R&D (pp. 218e227). Ber-lin/Heidelberg: Springer.

Brisaboa, N. R., Parama, J. R., Penabad, M. R., Places, A. S., &Rodrguez, F. J. (2002). Solving language problems in a multi-lingual digital library federation. In M. H. Shafazand, &A. M. Tjoa (Eds.), EurAsia-ICT 2002: Information and commu-nication technology (pp. 503e510). Berlin/Heidelberg:Springer.

Brisaboa, N. R., Penabad, M. R., Places, A. S., & Rodriguez, F. J.(2001). An architectural proposal for a cross-language systemto federate multilingual digital libraries. In Proceedings ofthe 3rd all-Russian scientific conference digital libraries:Promising advanced methods, techniques and technologies,electronic digital collections, Petrozavodsk, Russia,September 2001 (pp. 100e104). Retrieved from http://rcdl.ru/doc/2001/rodriguez.pdf.

Budzise-Weaver, T., Chen, J., & Mitchell, M. (2012). Collaborationand crowdsourcing: the cases of multilingual digital libraries.The Electronic Library, 30(2), 220e232. http://dx.doi.org/10.1108/02640471211221340.

CACAO. (2012). CACAO project: CACAO cross-language access tocatalogues and on-line libraries. Retrieved from http://www.cacaoproject.eu/.

Chen, C. C. (2001). Chinese Memory Net (CMNet): a model forcollaborative global digital library development. In C. ChenPlease cite this article in press as: Vassilakaki, E., & Garoufallou, E., Mand user-centered studies, information retrieval and user behavior,dx.doi.org/10.1016/j.iilr.2013.07.002nium: Fertile ground for distributed cross-disciplinary collab-oration (pp. 21e33). Beijing: Tsinghua University Press.

Chen, H. (2004a). Biomedical informatics and security informaticsresearch in digital library. In Z. Chen (Ed.), ICADL 2004 (pp.1e12). Berlin/Heidelberg: Springer.

Chen, H. (2004b). Digital library research in the US: an overviewwith a knowledge management perspective. Program: Elec-tronic Library and Information Systems, 38(3), 157e167. http://dx.doi.org/10.1108/00330330410547205.

Chen, H. H. (2004c). Multilingual information access in digital li-brary. In Proceedings of international conference on e-educa-tion 2004: Review and new perspectives (pp. 2e6). Retrievedfrom http://ntur.lib.ntu.edu.tw/retrieve/170757/58.pdf.

Chen, C. C. (2007). Delivery of web-based multilingual digital col-lections and services to multicultural populations: the case ofglobal memory net. In IFLA government libraries section, li-brary services to multicultural populations section at the IFLAmeeting in Durban, South Africa, August 18e24, 2007 (pp.1e11). Retrieved from http://www.ifla.org/IV/ifla73/Programme2007.htm.

Chen, J., & Bao, Y. (2009). Information access across languages onthe web: from search engines to digital libraries. Proceedings ofthe American Society for Information Science and Technology,46(1), 1e14. Retrieved from https://mail.asis.org/Conferences/AM09/contributedpapers/78.pdf.

Chen, A., & Gey, F. C. (2004). Multilingual Information Retrievalusing Machine Translation, relevance feedback and decom-pounding. Information Retrieval, 7(1e2), 149e182.

Cheng, K. S., Young, G. H., & Wong, K. F. (1999). A study on word-based and integral-bit Chinese text compression algorithms.Journal of the American Society for Information Science, 50(3),218e228.

Chen, J., & Ruiz, M. (2009). Towards an integrative approach to cross-language informationaccess fordigital libraries. In SIGIRworkshopon informationaccess inamultilingualworld July23, 2009Boston,Massachusetts USA (pp. 1e4). Retrieved from http://courses.unt.edu/ruiz/Publications/iaiNmw_unt-camera-ready.pdf.

Clinchant, S., & Renders, J. M. (2009). Multi-language models andmeta-dictionary adaptation for accessing multilingual digitallibraries. In C. Peters (Ed.), CLEF 2008 (pp. 83e88). Berlin/-Heidelberg: Springer.

Clough, P., & Eleta, I. (2010). Investigating language skills and fieldof knowledge on multilingual information access in digital li-braries. International Journal of Digital Libraries Systems, 1(1),89e103. Retrieved from http://ir.shef.ac.uk/cloughie/papers/IJDLS_2010.pdf.

Daranyi, S., Wittek, P., & Dobreva, M. (2010). Toward a 5M model ofdigital libraries. In Digital libraries for international develop-ment workshop (JCDL/ICADL 2010), Gold Coast, Australia (pp.3e13). Retrieved from http://eprints.port.ac.uk/4114/.

Dartois, M., Maeda, A., Fujita, T., Sakaguchi, T., Sugimoto, S., &Tabata, K. (1997). Building a multilingual electronic textcollection of folk tales as a set of encapsulated document ob-ject: an approach for casual users to browse multilingual doc-uments on the fly. In C. Peters, & C. Thanos (Eds.), Researchand advanced technology for digital libraries (pp. 215e231).Berlin/Heidelberg: Springer.

Di Nunzio, G. M., Leveling, J., & Mandl, T. (2011). Multilingual loganalysis: LogCLEF. In P. Clough, et al. (Eds.), ECIR 2011 (pp.675e678). Berlin/Heidelberg: Springer. Retrieved from http://doras.dcu.ie/16438/1/Multilingual_Log_Analysis_LogCLEF.pdf.

Diekema, A. R. (2012). Multilinguality in the digital library: a re-view. The Electronic Library, 30(2), 165e181. http://dx.doi.org/10.1108/02640471211221313.

Erich, A., Coman, M., & Trziman, E. (2010). Digital librariesinitiative in Romania. In Proceedings of the 9th WSEAS inter-ultilingual Digital Libraries: A review of issues in system-centeredThe International Information & Library Review (2013), http://-

multilingual digital libraries: a review of issues in system-centered and user-centered studies,...

Documents