MANAGING MULTIMODAL AND MULTILINGUALSEMANTIC CONTENT

Michael Martin1, Daniel Gerber2, Norman Heino1, Soren Auer1, Timofey Ermilov1

AKSW/Computer Science Institut, University of Leipzig, Postfach 100920, 04009 Leipzig, Germany1 {lastname}@informatik.uni-leipzig.de, 2 dgerber@informatik.uni-leipzig.de

Keywords: Knowledge Management, Semantic Web, Multimodality, Multilinguality, Semantic Wiki, LinkedData

Abstract: With the advent and increasing popularity of Semantic Wikis and the Linked Data the man-agement of semantically represented knowledge became mainstream. However, certain categoriesof semantically enriched content, such as multimodal documents as well as multilingual textualresources are still difficult to handle. In this paper, we present a comprehensive strategy formanaging the life-cycle of both multimodal and multilingual semantically enriched content. Thestrategy is based on extending a number of semantic knowledge management techniques such asauthoring, versioning, evolution, access and exploration for semantically enriched multimodal andmultilingual content. We showcase an implementation and user interface based on the semanticwiki paradigm and present a use case from the e-tourism domain.

1 INTRODUCTION

With the advent and increasing popularity ofSemantic Wikis and Linked Data the manage-ment of semantically represented knowledge be-came mainstream. Oracle, for example, inte-grated support for semantic knowledge manage-ment into their database product (Lopez and Das,2009), Google started to evaluate annotations1

using Resource Description Framework attributes(RDFa) and the W3C has lately launched thesecond revision of the Web Ontology Language(OWL) standard (Schneider, 2009). However, de-spite this progress certain categories of semanti-cally enriched content, such as multimodal doc-uments as well as multilingual textual resourcesare still difficult to handle.

Currently knowledge bases primarily containtyped data and a limited amount of textual con-tent, such as short labels, short descriptions orsmall hypertext fragments. With the increasingmaturity of semantic technologies and their wider

1http://googlewebmastercentral.blogspot.com/2009/05/introducing-rich-snippets.html

use in many different application scenarios therepresentation and interlinking of metadata formultimodal content such as audio, video, com-pound hypertext or multimedia documents is be-coming paramount. Another crucial feature of se-mantic knowledge representation is the languageindependence. Ontologies, taxonomies or simpleresource descriptions can be easily equipped withmultilingual texts and labels. However, the trans-lation and life-cycle of multilingual semantic con-tent is currently insufficiently supported.

Examples for the importance of supportingmultimodal and multilingual semantic contentare:

• Bio-medical semantic information systems. Inthe bio-medical domain vocabularies, tax-onomies and ontologies are already widelyused for structuring and aligning information,such as bio-medical knowledge bases (e. g. ondiseases and their symptoms), patient recordsand statistical data. Due to the time consum-ing creation and maintenance of these seman-tic artifacts and the increasing international-ization of the health care and life sciences do-

main, the translation and localization for dif-ferent languages becomes a key requirement.Also, the integration of multimodal content,such as depictions of disease symptoms, dia-grams as well as audio and video content isincreasingly important.

• Semantics based Web Content Management.RDF based knowledge representation strate-gies are increasingly used to manage Web con-tent on Web sites and Web-based informationsystems. The hypertext nature of the Webrequires the integration of different modali-ties, such as images, presentations, audio andvideo content. In addition, due to the globalaccessibility the availability of Web-based in-formation systems in different languages andlocalizations is a key requirement.

In this paper, we present a comprehensivestrategy for managing the life-cycle of both mul-timodal and multilingual semantically enrichedcontent. The strategy is based on extending anumber of semantic knowledge management tech-niques such as authoring, versioning, evolution,access and exploration for semantically enrichedmultimodal and multilingual content. With re-gard to multimedia content we devise a strat-egy for extracting, semantically representing andinterlinking metadata of multimedia documents.For the management of multilingual knowledgebases we developed techniques for supporting thelife-cycle of multilingual resources by enabling anefficient semi-automatic translation of individualproperty values, resources or all textual contentstored within a knowledge base. For keeping tex-tual content in a knowledge base in the preferredlanguage in sync with translations into other lan-guages we devise a strategy based on capitalizingthe integrated versioning of the Semantic DataWiki OntoWiki. We showcase an implementa-tion and user interface based on the semanticwiki paradigm and present a use case from thee-tourism domain.

The paper is structured as follows: We de-scribe a number of important aspects for man-aging semantic content in Section 2. We outlineour strategy for dealing with large quantities ofmultimodal content in Section 3. In Section 4 wepresent the strategy for supporting the life-cycleof multilingual resources. We showcase an appli-cation scenario in Section 5, where both – mul-timodal and multilingual – strategies for seman-tic content management are successfully applied.We review related work in Section 6 and concludewith an outlook on future work in Section 7

2 MANAGEMENT OFSEMANTIC CONTENT

The term semantic wiki is generally used for wikisystems that add additional semantic informationmanagement to classical text-based wiki systemsor allow management of structured semantic dataaccording to wiki principles (the latter often beingreferred to as (Semantic) Data Wikis).

One particular wiki system following the datawiki approach is OntoWiki (Auer et al., 2006). Itstarted as an RDF-based data wiki with empha-sis on collaboration but has meanwhile evolvedinto a comprehensive framework for developingSemantic Web applications (Heino et al., 2009).This involved not only the development of asophisticated extension interface allowing for awide range of customizations but also the addi-tion of several access and consumption interfacesallowing OntoWiki installations to play both aprovider and a consumer role in the emergingWeb of Data. In subsequent paragraphs wewill discuss several extensions to the OntoWikiframework that particularly facilitate authoringand management of multimodal and multilingualdata.

Application Layer

OntoWiki API, Access Interfaces Zend Framework

Persistence Layer

RDF Store

Stor

e Ad

apte

r

Authentication, ACL, Versioning, …

User Interface Layer

CSS Framework

OntoWiki UI API RDFauthor Templates

Extensions (Evolution,

Multimedia, …)

Figure 1: Overview of OntoWiki’s architecture withextension API and Zend web framework (mod. ac-cording to (Heino et al., 2009)).

2.1 Authoring

Semantic content in OntoWiki is represented asresource descriptions. Following the RDF datamodel representing one of the foundations of the

Semantic Web vision, resource descriptions arerepresented (at the lowest level) in the formof statements. Each of these statements (ortriples) consist of a subject which identifies a re-source as well as a predicate and an object whichtogether represent data about said resource ina fashion reminiscent of key-value pairs. Bymeans of RDFa2, these statements are retainedin the HTML view (i.e. user interface) part andare thus accessible to client-side techniques likeJavaScript.

Authoring of such content is based on saidclient-side representation by employing the RD-Fauthor approach (Tramp et al., 2010b): viewsare declared in terms of the model language(RDF) which allows the underlying model be re-stored. Based on this model, a user interface canbe generated with the model being providing allthe domain knowledge required to do so. TheRDFauthor system provides an extensible set ofauthoring widgets specialized for certain editingtasks. In the work at hand, we extended thesystem by adding capabilities for automaticallytranslating literal object values. Since the seman-tic context is known to the system, these transla-tion functionality can be bound to arbitrary char-acteristics of the data (e. g. to a certain propertyor a missing language).

2.2 Versioning

As outlined in the wiki principles, keeping trackof all changes is an important task in order toencourage user participation. OntoWiki appliesthis concept to RDF-based knowledge engineeringin that all changes are tracked on the statementlevel (Auer and Herre, 2006). These low-levelchanges can be grouped to reflect application-and domain-specific tasks involving modificationsto several statements as a single versioned item.Provenance information as well as other meta-data (such as time, user or context) of a particu-lar changeset can be attached to each individualchangeset. All changes on the knowledge base canbe easily reviewed and rolled-back if needed.

2.3 Evolution

The loosely typed data model of RDF encouragescontinuous evolution and refinement of knowledgebases. With EvoPat, OntoWiki supports this ina declarative, pattern-based manner (Rieß et al.,

2http://www.w3.org/TR/rdfa-syntax/

2010). Basic evolution patterns consist of threecomponents:

• a set of variables,

• a SPARQL select query selecting a number ofresources under evolution,

• a SPARQL/Update query template that is ex-ecuted for each resulting resource of the selectquery.

In addition, basic patterns can be combined toform compound patterns—suitable for more com-plex evolution scenarios.

In order to facilitate the semi-automatic ap-plication of evolution patterns, bad smells can bedefined that serve as a detection mechanism forontology design anti-patterns or data modelingproblems. If certain conditions are met, this pro-cess is even fully automatable.

2.4 Access Interfaces

In addition to human-targeted graphical userinterfaces, OntoWiki supports a number ofmachine-accessible data interfaces. These arebased on established Semantic Web standards likeSPARQL or accepted best practices like publica-tion and consumption of Linked Data.

SPARQL Endpoint. The SPARQL recom-mendation not only defines a query language forRDF but also a protocol for sending queries toand receiving results from remote endpoints3.OntoWiki implements this specification, allow-ing all resources managed in an OntoWiki bequeried over the Web. In fact, the aforemen-tioned RDFauthor authoring interface makes useof SPARQL to query for additional schema-related information, treating OntoWiki as a re-mote endpoint in that case.

Linked Data. Each OntoWiki installation canbe part of the emerging Linked Data Web. Ac-cording to accepted publication principles 4, On-toWiki makes all resources accessible by its URI(provided, the resource’s URI is in the samenamespace as the OntoWiki instance). Further-more, for each resource used in OntoWiki ad-ditional triples can be fetches if the resource isdereferenceable.

3http://www.w3.org/TR/rdf-sparql-protocol/4http://sites.wiwiss.fu-berlin.de/suhl/

bizer/pub/LinkedDataTutorial/

Semantic Pingback. Pingback is an estab-lished notification system that gained wide popu-larity in the blogsphere. With Semantic Ping-back (Tramp et al., 2010a), OntoWiki adaptsthis idea to Linked Data providing a notificationmechanism for resource usage. If a Pingback-enabled resource is mentioned (i. e. linked to) byanother party, its pingback server is notified ofthe usage. Provided, the Semantic Pingback ex-tension is enabled all resources used in OntoWikiare pinged automatically and all resources definedin OntoWiki are Pingback-enabled.

2.5 Exploration Interfaces

For exploring semantic content, OntoWiki pro-vides several exploration interfaces that rangefrom generic views over search interfaces to so-phisticated querying capabilities for more RDF-knowledgable users. The subsequent paragraphsgive an overview of each of them.

Knowledge base as an information map.The compromise of, on the one hand, providinga generic user interface for arbitrary RDF knowl-edge bases and, on the other hand, aiming at be-ing as intuitive as possible is tackled by regard-ing knowledge bases as information maps. Eachnode at the information map, i. e. RDF resource,is represented as a Web accessible page and in-terlinked to related digital resources. These Webpages representing nodes in the information mapare divided into three parts: a left sidebar, a maincontent section and a right sidebar. The left side-bar offers the selection of content to display inthe main content section. Selection opportuni-ties include the set of available knowledge bases,a hierarchical browser and a full-text search.

Full-text search. The full-text search makesuse of special indexes (mapped to proprietarySPARQL syntax) if the underlying knowledgestore provides this feature, else, plain SPARQLstring matching is used. In both cases, the result-ing SPARQL query is stored as an object whichcan later be modified (e. g. have its filter clausesrefined). Thus, full-text search is seamlessly inte-grated with facet-based browsing (see below).

Content specific browsing interfaces. Fordomain-specific use cases, OntoWiki provides aneasy-to-use extension interface that enables theintegration of custom components. By providingsuch a custom view, it is even possible to hide

completely the fact that an RDF knowledge baseis worked on. This permits OntoWiki to be usedas a data-entry frontend for users with a less pro-found knowledge of Semantic Web technologies.

Faceted-browsing. Via its facet-based brows-ing, OntoWiki allows the construction of complexconcept definitions, with a pre-defined class as astarting point by means of property value restric-tions. These two views are sufficient for browsingand editing all information contained in a knowl-edge base in a generic way.

Query-builder. OntoWiki serves as aSPARQL endpoint, however, it quickly turnedout that formulating SPARQL queries is tootedious for end users. In order to simplify thecreation of queries, we developed the VisualQuery Builder5 (VQB) as an OntoWiki exten-sion, which is implemented in JavaScript andcommunicates with the triple store using theSPARQL language and protocol. VQB allows tovisually create queries to the stored knowledgebase and supports domain experts with anintuitive visual representation of query and data.Developed queries can be stored and added viadrag-and-drop to the current query. This enablesthe reuse of existing queries as building blocksfor more complex ones.

3 MULTIMODAL SEMANTICCONTENT

To handle large amounts of multimedia data, au-tomatic processes for managing this kind of con-tent have to be developed. To fulfill this need weimplemented a PHP-framework based on Erfurt6,which has been integrated into OntoWiki. Withthe help of this framework, it is possible to importarbitrary multimedia documents (support for 13different file types is currently implemented) oreven complete directory structures into a knowl-edge base and manage them accordingly with On-toWiki, using the techniques presented in Sec-tion 2. The workflow for importing multimediadocuments into OntoWiki is presented in Figure 2and described in detail in the subsequent sections.

5http://aksw.org/Projects/OntoWiki/Extension/VQB

6http://aksw.org/Projects/Erfurt

Discogs, Flickr, MusicBrainz...

literal valueliteral value

literal value literal valuequery

query

response

responsesave

MultimediaVocabulary

define

Input Extraction Representation Linking

? ? ? ?

Metadata

Figure 2: Birds eye view on the process of multimedia metadata extraction, representation and interlinking.

3.1 Extracting MultimediaMetadata

The variety of multimedia document types is verylarge. For example, there are more then 1000 dif-ferent MIME types registered at IANA7. Due tothat, it is not possible to create a generic extrac-tion mechanism, which works for all multimediafile types. For this reason we developed a frame-work, which detects certain formats and is able toreact correspondingly. Furthermore, it is highlyconfigurable and easily extensible. For instance,it is possible to integrate support for new multi-media types and to configure the properties andclasses used to create the semantic representationvia various configuration options. The extractionof multimedia metadata is realized in the follow-ing steps:

1. Extraction of general metadata attributes.Every file, regardless of its type, can be de-scribed by a number of general attributes.Information about the file’s name, size ordate of creation may be extracted for eachfile. This information can be extracted evenwithout the knowledge of the file’s type andcan therefore be processed identically for eachfile.7http://www.iana.org/assignments/

media-types/

2. Extraction of specialized metadata attributes.In addition to general information, manymultimedia formats already contain metadataspecific to their field of use. Such informationis most likely arranged in key-values pairs inthe file’s header. For instance, music files dousually contain ID3 tags8 and images takenby digital cameras include EXIF (JapanElectronics and Information TechnologyIndustries Association, 2002) information.For the extraction of this metadata, frame-works like getID3 9 and Zend10 are used.However, it is necessary to determine thefile’s type in order to start a specific ex-traction mechanism. The MIME type ofthe file is determined and subsequently aspecialized metadata extractor is initialized.The framework is designed in such a waythat every metadata extractor manages a setof extensions, each one being responsible forthe extraction of a single metadata type onits own. These extensions will be executedconsecutively, thus giving the opportunityto re-use already extracted metadata andaccelerate the extraction process.Especially when dealing with large data

8http://www.id3.org/9http://getid3.sourceforge.net/

10http://framework.zend.com/

sets or working on mobile devices (withlow storage capacity and bandwidth), a filepreview thumbnail is an important metadataitem. For this reason, previews of PDF orvideo documents are created with the helpof the tools convert11 and ffmpeg12. In thiscontext we also developed an extensiblePreview Module for OntoWiki, which allowsto view those generated previews for variousdocument formats.Other examples of such metadata extractionextensions may be the number of pages of aPDF document or the geo-coordinates of animage.

3. Integration of additional information.In order to actually take advantage ofRDF and Linked Data to discover similarresources, it may not be useful to extractthe artists name only as a literal value. Inthe third step of this extraction process thepreviously extracted metadata is now usedto integrate additional information, which isnot explicitly contained in the processed files.For example, an artists name extracted fromthe music’s file ID3 information may be usedto look up a URI for this artist in the LinkedData Cloud. Likewise, traditional non-RDFbased web-services may be used to gatheradditional information like the cover of thecorresponding album of a song. The use ofURIs for certain concepts (album, artist etc.)allows to integrate more data with the helpof OntoWiki’s Linked Data import functionlater in the authoring process.

3.2 Representing MultimediaMetadata

To represent the extracted information in RDFwe did not create new vocabularies, but re-used many already well established ones. Anoverview of those vocabularies used in the dif-ferent areas of multimedia is depicted in Fig-ure 3. The main concept of this repre-sentation is the separation between the ac-tual data (nfo:FileDataObject) and its inter-pretation (nie:InformationElement) borrowedfrom the NEPOMUK information element on-tology (NIE) and the NEPOMUK file ontol-ogy (NFO)13. The large number of subclasses of

11http://www.imagemagick.org12http://www.ffmpeg.org13NIE, NOE and NEXIF are available from: http:

//www.semanticdesktop.org/ontologies/

nie:InformationElement makes it possible toclassify most of the common multimedia types.The separation between the two concepts orig-inates from the fact that the representationof the data is usually not of interest for theuser. To comprehend a DataObject neverthe-less, it needs to be interpreted as a correspondingInformationElement or as one of its subclasses.This is achieved with the properties nie:hasPartor its inverse counterpart nie:isPartOf. Thisapproach will ensure that even complex datastructures, like archives in the attachment ofemails are processed correctly.

In order to describe the individualInformationElements further, the followinglist of vocabularies has been chosen:

• For describing audio documents the music on-tology (Raimond et al., 2007) is used (withnamespace prefix mo), since it allows to repre-sent all information available in ID3. Beyondthat, also concepts like concert or festival arerepresented.

• For describing PDF documents propertiesand classes of various vocabularies such asDublin Core14 (namespace prefix dc), NFOand RDFS are used, since no single vocab-ulary was found which is able to represent allinformation.

• For describing image documents the NEPO-MUK EXIF ontology is used, since therdfs:range and rdfs:domain for those prop-erties are set as opposed to Kanzaki’s or theW3C EXIF ontologies. Additionally, the rep-resentation of all properties used in EXIF ispossible with the NEPOMUK EXIF ontology.

• Due to the fact, that those types of video doc-uments supported by the framework do origi-nally not contain any additional metadata, itis only possible to extract low-level informa-tion like the video’s frame-rate or the usedvideo codec. Therefore, the use of heavy-weight ontologies like COMM15 is discour-aged. All extractable information is repre-sentable using the NEPOMUK file ontology.

• The URIs used to describe the file’s MIMEtype are provided by the mediatypes16 appli-cation, which routinely scrapes all registeredMIME types registered at IANA.

14http://dublincore.org/15http://comm.semanticweb.org/16http://mediatypes.appspot.com

http://purl.org/NET/mediatype#MediaType

audio/mpegapplication/

pdf

nfo:FileDataObject

nfo:Folder

nfo:FileHash

nfo:PaginatedTextDocument

mo:AudioFile

mo:Trackmo:MusicArtist

mo:Genre

mo:Record

nfo:Image nfo:Video

nie:InformationElement

nie:hasPart

rdf:typerdf:type rdf:type rdf:type

rdfs:subClassOfrdfs:subClassOfrdfs:subClassOf

rdf:type

mo:discogsmo:track_numbermo:release_typemo:image

dc:titledc:creatordc:subjectdc:createddc:modifiedpdf:Keywordspdf:Producerrdfs:labelrdfs:seeAlsonfo:pageCountvm:hasPreview

nfo:permissionsnfo:fileLastAccessednfo:fileLastModifiednfo:fileName

dc:creatormo:track

mo:genremo:available_as

nfo:Audio rdfs:subClassOf

rdf:type

audio/ogg

rdf:type

image/jpeg image/png

rdf:type

video/x-msvideo

video/mpeg

nfo:durationnfo:codecnfo:frameRatenfo:frameCountnfo:heightnfo:widthnfo:channelsnfo:sampleRate

nexif:imageWidthnexif:imageLengthnexif:flashnexif:isoSpeedRatingsnexif:exposureTimenexif:gpsLatitudenexif:gpsLongitudenexif:modelnfo:horizontalResolutionnfo:verticalResolution

rr

rr

d d

d d

nfo:fileSizenfo:fileUrlnfo:isLogicalPartOfnfo:hasHash

Figure 3: Mashup of different vocabularies used to represent multimedia metadata in RDF.

3.3 Interlinking Metadata

The extracted and in RDF represented metainfor-mation of the multimedia documents can now beused to find and create links between those andarbitrary other resources. For this reason, we de-veloped the OntoWiki Linking Module, which isable to discover possible links between any sortof resources, inparticular multimedia documents.The semi-automatic algorithm used to locate theresources for linking can be divided in the fol-lowing four parts, taking into account that thestarting point of this algorithm is an arbitraryresource r.

1. Find all properties with the rdfs:rangeor rdfs:domain of the type (rdf:type) ofressource r.

2. Afterwards a list of all resources is cre-ated, which are accessible via those proper-ties collected in step one. This means, ifa property was found in step one with therdfs:range/rdfs:domain of the type of r, allresources are listed, which match the type ofthe rdfs:domain/rdfs:range restriction ofthis property. In future versions of this algo-rithm this might also be extended to includesuper classes of r and of the correspondingrdfs:range or rdfs:domain restrictions.

3. All resources found in step two are now com-pared to r. The comparison takes place

with the help of textual attributes, usuallydc:title or rdfs:label, of those resources.The metric to calculate the probability of in-terlinking is the effectiveness measure (van Ri-jsbergen, 1979), the weighted harmonic meanbetween precision and recall:

E = 1α( 1

p )+(1−α) 1r

We are aware of the fact, that some resourcesmay not have any textual descriptions and aretherefore not suitable for this kind of compar-ison. Nevertheless we choose this similarityanalysis because we consider textual descrip-tions as the smallest common attribute andthus make it possible to compare the majorityof RDF resources. To improve this compari-son further, we plan to implement differentalgorithms. For instance, it is reasonable tolink resources whose geographical distance isbelow a certain threshold.

4. The found resources are now sorted by theirprobability of linking to r and presented tothe user (grouped by the property). Thus theuser is able to review, approve and establishthe links. An evaluation of this algorithm hasshown that this linking could also be donecompletely automatically, if the probability isabove a certain threshold.

We developed two vocabularies which arealigned to the different MIME type categories, for

linking multimedia documents with arbitrary re-sources in step four. The more generic one, shownin excerpts in Listing 1, consists of the prop-erty mm:hasMultimediaDocument and five sub-properties, which correspond to the five mainMIME types (application, audio, video, text andimage). Those properties may now be used to linkarbitrary resources. However, for creating morespecific statements, users of this vocabulary areadvised to extend this vocabulary (cf. Section 5).

@prefix vm: <http :// vakantieland.nl/multimedia/> .

@prefix mm: <http ://ns.aksw.org/mm/> .

mm:hasMultimediaDocument

rdf:type owl:ObjectProperty ;

rdfs:range nie:InformationElement ;

rdfs:domain rdfs:Resource ;

rdfs:label "has multimedia document" .

mm:hasAudioDocument

rdfs:subPropertyOf mm:hasMultimediaDocument ;

rdfs:range nfo:Audio ;

rdfs:label "has audio document" .

Listing 1: Excerpt of the generic multimedia docu-ment and arbitrary resources interlinking ontology.

4 MULTILINGUAL SEMANTICCONTENT

The overall life-cycle of multilingual semanticcontent in knowledge bases is depicted in Fig-ure 4. The process usually starts with the cre-ation and authoring of a semantic resource. Oncecreated textual content can be translated. Sub-sequently, the original language content attachedto the resource might be revised, which has totrigger a revision of the translations as well.

Figure 4: Life-cycle of multilingual resources.

To increase the comprehensibility of RDFknowledge bases for users and to support multi-linguality efficiently for Semantic Web Applica-tions (Martin and Auer, 2010) based on RDFknowledge bases, RDF resources should be la-beled and commented in multiple languages asexemplary illustrated in Listing 2.

geo:Netherlands rdfs:label "Netherlands"@en

geo:Netherlands rdfs:label "Niederlande"@de

Listing 2: RDF resource with labels in different lan-guages.

Most RDF resources contain at least one label ina preferred language, that does not have alwaysto be defined using a language tag in conjunctionwith the literal value (e. g. string@lang). In or-der to support multi-linguality of RDF knowledgebases, content authors have to translate and storethese literal values into other languages to be sup-ported. To assist the work-flow of translating andstoring RDF literal values in a (semi-)automatedway, we developed a set of OntoWiki extensionsfor language resource translation and manage-ment:

1. RDFauthor extension, for translating singleRDF literal values,

2. Individual resource translation module, fortranslating all string literal values attached toa certain resource,

3. Massive translation component, for translat-ing literal values of a complete knowledgebase.

4. Multi-lingual resource versioning and revision,for notifying content authors to revise newtranslations.

To enable the semi-automated translation ofliteral values we employ the Google TranslationService with its API17. This service supports thetranslation between more than 50 languages suchas English, Russian, German, Greek, Vietnamese,Hindi etc. Due to the fact that RDF literal val-ues, which have to be translated, do not alwaysexplicitly contain a language tag, it might be re-quired to detect this language code automatically.The Google Translate API support this function-ality. Since not all of possible languages have tobe supported by a certain knowledge base, con-tent authors are able to configure a set of thedesired languages.

Moreover, not all literal values contain trans-latable information. Due to that fact, the devel-oped extensions are configurable in order to selectdesired property URIs used to attach translatableliteral values. In case that resources still containa set of translated literal values, the user is ableto select a preferred language, which is being usedto select the literal value as source for the transla-tion. In the next paragraphs we describe the On-

17Google Translate API: http://code.google.com/apis/ajaxlanguage/

toWiki extensions for language resource transla-tion and management we developed in some moredetail.

RDFauthor extension. As being mentionedin Section 2.1, it is possible to integrate widgetsinto RDFauthor. These can be used to displayand process values for configured properties in aspecial manner, or furthermore to support specialworkflows. By default, the user is able to manu-ally encode for a selected property multiple literalvalues and in addition a data-type or languagetag. To support a (semi-)automated translationof single literal values while creating or maintain-ing RDF resources in RDFauthor, we created awidget as depicted in Figure 5. After adding or

Figure 5: Adding literal values using RDFauthor andtranslation support.

editing a literal value with this RDFauthor wid-get, the user is able to duplicate newly added oredited information represented in the widget. Af-ter selecting a different language for that dupli-cated information in the GUI, the Google Trans-lation Service is called and the literal value istranslated automatically. Finally, the user is ableto revise and improve the suggested translationresult.

Individual resource translation module.For the more automated and less time-consumingtranslation of individual resources, which containat least one translatable literal value, we devel-oped the individual resource translation moduledepicted in Figure 6. The content author canconfigure a set of properties, a set of to be sup-ported languages and a single preferred languageat a particular time. After selecting a resource inOntoWiki, this extension notifies the user, that

not all translations for the set of configured pred-icates exist yet. Then the user only has to trig-ger the translation functionality. The algorithmqueries the knowledge base obtaining all proper-ties and assigned literal values for the selectedresource. Then the translation algorithm selectsfor every configured property one literal value (ifexisting in the preferred language) and requestsmissing translations for every configured languagefrom the translation service. The resulting trans-lations are stored immediately and presented tothe user, where she is able to check the quality ofthe translations. Unacceptable translations canalso be changed manually by using RDFauthor.

Massive translation component. In order totranslate RDF resources massively, we developedthe Massive translation component for OntoWiki,which is operating similar to the Individual re-source translation module. By using SPARQL,this component requests a set of resources withmissing translations. The result is fulfilled auto-matically by using the translation service and pre-sented to the user as a HTML-form in the GUI asillustrated in figure 7. All literal values in the gen-

Figure 7: Adding translated literal values to selectedknowledge base semi-automatically

erated HTML-form are editable, due to improvetranslations manually afterwards. After applyingand saving the new translations, this algorithmis looped until all resources contain the specifiedamount of translated literal values.

Figure 6: Adding translated literal values to selected resource automatically

Multi-lingual resource versioning and re-vision. Every change between its creation anddeletion of an RDF resource is tracked by On-toWikis versioning component described in Sec-tion 2.2. This versioning component is also usedto store information about the translation pro-cess. After changing one of the literal values,translations to other languages could be affected.As a consequence, a special entry marking atranslation process is stored to the versioningrepository. As being depicted in Figure 4, everytranslation might have to be revised. This flagis used to notify the content author for approv-ing the correctness of all other translated literalvalues of the particular property of the selectedRDF resource. After approving the correctness afurther flag is stored to represent an acceptabletranslation state of the RDF resource.

5 USE CASE: VAKANTIELAND

Both our semantic content management strate-gies were applied and evaluated in the SemanticWeb application Vakantieland18. Vakantielandpublishes comprehensive information about20,000 touristic points-of-interest (POI) inthe Netherlands such as textual descriptions,location information and opening hours. Theinformation is stored in a knowledge base con-taining almost 2 million triples and is structuredusing approximately 1,250 properties as well as400 classes. Vakantieland was designed accordingto the model/view/controller principle and uses

18Available at: http://staging.vakantieland.nl

the Erfurt API as middleware, which is alsoused in OntoWiki. Almost all of the informationpresented in Vakantieland is retrieved usingSPARQL.

Multimedia management in Vakantieland.In this use case we applied the multimedia man-agement process, presented in Section 3, to createRDF resources for about 850 PDF documents (i.e.info brochures of POIs) and interlink them ac-cordingly. In particular, we extended the genericmultimedia linking vocabulary as shown in List-ing 3, in order to specify the rdfs:domain toPOIs and evaluated the OntoWiki Linking Mod-ule. For one hundred randomly chosen documentsthe suggestions of this module have been com-pared to manually assigned links, created by a do-main expert. This evaluation has shown, that for80% of the documents, the correct suggestion –the POI with the highest probability – was found.The other way around, i.e. suggest documentsfor POIs, it was even possible to find the cor-rect one in 90% of the examined cases. The linkscreated this way are then used to display thosedocuments, or any other document type for thatmatter, and additional information like the docu-ment’s title (translated in various languages), ona POI’s details page.

@prefix vm: <http :// vakantieland.nl/multimedia/> .

@prefix mm: <http ://ns.aksw.org/mm/> .

vm:hasApplicationDocument

rdfs:subPropertyOf mm:hasApplicationDocument ;

rdfs:domain vak:POI ;

rdfs:label "POI has an application document ".

Listing 3: Excerpt of the document interlinking on-tology used in the Vakantieland use case.

Figure 8: Excerpt of the Vakantieland user interface with translated categories.

Multilinguality management inVakantieland. In order to present the tourismcontent of Vakantieland in multiple languages,we encoded translations of class labels, propertylabels and, if possible, also property values inRDF as described in Section 4. To support theprocess of translating resources, we used thesketched OntoWiki modules, to decrease thetranslation time consumed. Several componentsof the Vakantieland GUI use these translationssuch as the category selector on top of thepage or the faceted category filter located atthe right sidebar of the page as depicted inFigure 8. At this time, the tourism RDF contentof Vakantieland contains information encodedin various different European languages such asDutch, English, French, German, Italian andSpanish. With the help of the mentioned On-toWiki multilinguality modules, it is possible toencode literal values in more than 50 languages,which will be done when Vakantieland is adoptedto further countries.

6 RELATED WORK

The wide-spread use of Internet has forced cre-ation of multilingual web sites to allow usage byanyone on the Web. At the moment there are alot of tools available for fast translation of almostany web sites, articles or resources. For exam-

ple, Google Translate19 allows not only transla-tion of web resource and uploaded texts, but alsoan on-fly translation of pages for search resultsin Google. There are also quite some plugins forCMS and Wiki systems like JA Translate20 forJoomla, WikiBhasha21 for Wikipedia or GlobalTranslator22 for Wordpress.

The same is now happening in the Seman-tic Web – due to increase in Semantic Web re-search activity on a worldwide scale and sincemost ontologies are developed in a single lan-guage (mostly in English), an aspiration in on-tology localization appeared. Though there isa large amount of literature on topic of ontol-ogy multilinguality and localization, most of them(e. g. (Montiel-Ponsoda et al., 2008)) are focusedon multilingual ontology modeling, not on local-ization process itself or tools used for it. Oneprominent example in that context is also Word-net (Fellbaum, 2010), which constitutes a lexicaldatabases which is useful in process of transla-tion but only as an additional resource. A study,which evaluates various Semantic Web standardsand technologies for their readiness for interna-tionalization is presented in (Auer et al., 2010).

19http://googlesystem.blogspot.com/2010/08/how-google-translate-works.html

20http://www.joomlart.com/joomla/extensions/ja-google-translate

21http://www.wikibhasha.org/22http://wordpress.org/extend/plugins/global-

translator/

Basically all those publications investigate how todesign ontologies containing several languages foreach resource, but say nothing about the transla-tion and multilingual content management pro-cess required to obtain truly internationalizedknowledge bases.

There are only a few papers about ontologylocalization tools. The LabelTranslator was de-scribed in (Espinoza et al., 2008) and later in(Espinoza et al., 2009) as a more advanced, com-plex system for translation. Though, this systemonly works for labels and supports only 3 lan-guages: English, German and Spanish. All ofthe described approaches use additional models(e. g. LIR in (Montiel-Ponsoda et al., 2008)) forstoring information in additional languages. Thiswill produce a large number of additional triples,which will slow down the overall performance ofknowledge base if it is largely sized. There’salso tools like LOD In Translation23 that providetranslation based on existing data in LOD cloudand return results as set of URIs for resources, butthe downside of this tool is that it provides onlystring literal translation and so cannot be appliedto semantic resources as it is. That’s why we de-velop a new approach that uses RDF languagetags and only produces one triple for each lan-guage, and thus will scale much better for largerknowledge bases.

Most of the published papers about multime-dia content annotation using ontologies are fo-cused on human-driven approaches for file anno-tation. Also, often existing approaches are fo-cused on only one type of file, e. g. annotatingphotos as described in (Schreiber et al., 2001) oron specific annotating vocabularies, e. g. OntoE-LAN linguistic annotation tool as presented in(Chebotko et al., 2004). There are also broaderapproaches oriented on converting multimediacontent description standards (e. g. MPEG-7)to domain specific concepts like presented in(Athanasiadis et al., 2005). To the best of ourknowledge, none of published approaches usesautomated multimedia content annotation usingwide variety of standardized vocabularies.

7 CONCLUSIONS

With the increasing maturation of semantic tech-nologies the facilitation of multimodal and mul-tilingual semantic content management became a

23http://semlabs.upmc.fr/LODInTranslation/

crucial requirement. In this article we presentedtwo complementary strategies for managing mul-timodal and multilingual semantic content basedon the semantic wiki paradigm. Both strategiesare based on supporting the lifecycle of respectivesemantic content.

With regard to future work we see in particu-lar the following directions:

Integration of automatic linking tech-niques. Establishing and maintaining links onthe Web of Data is still a major challenge. Withregard to multi-media data we envision the re-alization, of a linking dashboard, with pluggablelinking services, which particularly facilitate thelinking of local multimedia assets based on theextracted meta-data with resources available onthe Web of Data.

Fine-grained provenance tracking. Alreadynow basic provenance information such as the ed-itor of a certain translation or multimedia anno-tation is tracked by OntoWiki. However, we en-vision a more fine-grained representation, whichincludes information about the employed tools(such as multimedia metadate extraction, auto-matic translation etc.) in order to facilitate fu-ture revisions (e.g. based on new tool releasesetc.)

Facilitation of adaptive previews. An cru-cial component of multimodal information sys-tems are previews of relevant (parts of the) mul-timedia assets. We aim at integrating previewfacilities, which take the users’s context (such aslocality, search and exploration history, interestsetc.) into account.

ACKNOWLEDGEMENTS

We would like to thank the LOD2 consor-tium (http://lod2.eu) for the helpful com-ments and inspiring discussions during the workdescribed in this article. The research lead-ing to these results has received funding fromthe European Union Seventh Framework Pro-gramme (FP7/2007-2013) under grant agreementno. 257943.

REFERENCES

Athanasiadis, T., Tzouvaras, V., Petridis, K., Pre-cioso, F., Avrithis, Y., and Kompatsiaris, Y.(2005). Using a multimedia ontology infras-tructure for semantic annotation of multimediacontent. In in Proceedgins of 5th InternationalWorkshop on Knowledge Markup and SemanticAnnotation (SemAnnot 05.

Auer, S., Dietzold, S., and Riechert, T. (2006). On-toWiki - A Tool for Social, Semantic Collabo-ration. In Cruz, I. F., Decker, S., Allemang,D., Preist, C., Schwabe, D., Mika, P., Uschold,M., and Aroyo, L., editors, The Semantic Web- ISWC 2006, 5th International Semantic, vol-ume 4273 of Lecture Notes in Computer Science,pages 736–749. Springer.

Auer, S. and Herre, H. (2006). A versioning and evo-lution framework for rdf knowledge bases. InProceedings of at Sixth International Andrei Er-shov Memorial Conference - Perspectives of Sys-tem Informatics (PSI06), 27-30 June, Novosi-birsk, Akademgorodok, Russia, volume 4378.

Auer, S., Weidl, M., Lehmann, J., Zaveri, A. J., andChoi, K.-S. (2010). I18n of semantic web appli-cations. In Proceedings of the 9th InternationalSemantic Web Conference (ISWC2010), LectureNotes in Computer Science, Berlin / Heidelberg.Springer.

Chebotko, A., Deng, Y., Lu, S., Fotouhi, F., Aristar,A., Brugman, H., Klassmann, A., Sloetjes, H.,Russel, A., and Wittenburg, P. (2004). Ontoe-lan. an ontology-based linguistic multimedia an-notator. Multimedia Software Engineering, In-ternational Symposium on, 0:329–336.

Espinoza, M., Gmez-Prez, A., and Montiel-Ponsoda,E. (2009). Multilingual and localization sup-port for ontologies. In Aroyo, L., Traverso, P.,Ciravegna, F., Cimiano, P., Heath, T., Hyvnen,E., Mizoguchi, R., Oren, E., Sabou, M., andSimperl, E., editors, The Semantic Web: Re-search and Applications, volume 5554 of LNCS,pages 821–825. Springer.

Espinoza, M., Gomez-Perez, A., and Mena, E. (2008).Labeltranslator - a tool to automatically localizean ontology. In ESWC’08: Proceedings of the 5thEuropean semantic web conference on The se-mantic web, pages 792–796, Berlin, Heidelberg.Springer.

Fellbaum, C. (2010). Wordnet. In Poli, R., Healy,M., and Kameas, A., editors, Theory and Ap-plications of Ontology: Computer Applications,pages 231–243. Springer Netherlands.

Heino, N., Dietzold, S., Martin, M., and Auer, S.(2009). Developing semantic web applicationswith the ontowiki framework. In Pellegrini, T.,Auer, S., Tochtermann, K., and Schaffert, S., ed-itors, Networked Knowledge - Networked Media,volume 221 of Studies in Computational Intel-ligence, pages 61–77. Springer, Berlin / Heidel-berg.

Japan Electronics and Information Technology Indus-tries Association (2002). Exchangeable image fileformat for digital still cameras: Exif version 2.2.JEITA CP-3451.

Lopez, X. and Das, S. (2009). Oracle database 11gsemantic technologies, semantic data integrationfor the enterprise, white paper. Technical report,Oracle Semantic Technologies Center.

Martin, M. and Auer, S. (2010). Categorisation of Se-mantic Web Applications. In proceedings of the4th International Conference on Advances in Se-mantic Processing (SEMAPRO2010) 25 October– 30 October, Florence, Italy.

Montiel-Ponsoda, E., de Cea, G. A., Gmez-Prez, A.,and Peters, W. (2008). Modelling multilingualityin ontologies. Poster, CoLing 2008.

Raimond, Y., Abdallah, S. A., Sandler, M., and Gi-asson, F. (2007). The music ontology. In Pro-ceedings of the 8th International Conference onMusic Information Retrieval (ISMIR 2007).

Rieß, C., Heino, N., Tramp, S., and Auer, S. (2010).Evopat – pattern-based evolution and refactor-ing of rdf knowledge bases. In Proceedings ofthe 9th International Semantic Web Conference(ISWC2010), Lecture Notes in Computer Sci-ence, Berlin / Heidelberg. Springer.

Schneider, M. (2009). Owl 2 web ontology languagerdf-based semantics. Technical report, W3CRec.

Schreiber, A. T. G., Dubbeldam, B., Wielemaker, J.,and Wielinga, B. (2001). Ontology-based photoannotation. IEEE Intelligent Systems, 16:66–74.

Tramp, S., Frischmuth, P., Ermilov, T., and Auer,S. (2010a). Weaving a social data web with se-mantic pingback. In Cimiano, P. and Pinto, H.,editors, Proceedings of the EKAW 2010 - Knowl-edge Engineering and Knowledge Managementby the Masses; 11th October-15th October 2010 -Lisbon, Portugal, volume 6317 of Lecture Notesin Artificial Intelligence (LNAI), pages 135–149,Berlin / Heidelberg. Springer.

Tramp, S., Heino, N., Auer, S., and Frischmuth, P.(2010b). Rdfauthor: Employing rdfa for collab-orative knowledge engineering. In Cimiano, P.and Pinto, H., editors, Proceedings of the EKAW2010 - Knowledge Engineering and KnowledgeManagement by the Masses; 11th October-15thOctober 2010 - Lisbon, Portugal, volume 6317 ofLecture Notes in Artificial Intelligence (LNAI),pages 90–104, Berlin / Heidelberg. Springer.

van Rijsbergen, C. J. (1979). Information Retrieval.Butterworths, London.