semantic search using rdf metadata (semtech 2005)

Semantic Search Using RDF Metadata

Semantic Technology Conference 2005 8 March 2005

Bradley P. AllenSiderean Software, Inc.

Copyright © 2005 Siderean Software, Inc. All rights reserved.

Overview

• Semantic search• Motivation

• Enterprise adoption

• Semantic search using RDF• Examples

• Lessons

• Directions


Problem• “We have to understand what information we

have and organize it,’ says [Santa Clara Co. CIO] Ajmani, who estimates that saving each employee an hour a month spent looking for information would save millions of dollars.” [Information Week, 1/19/04]

• “…typical enterprise floundering in a sea of information … too many repositories, each with its own set of applications.” [IDC, 2004]

• “The search capabilities on most company and content-oriented Web sites are as bad now as they were several years ago.” [eWeek, 1/26/04]


Portal-driven demand for a better solution

• “A portal provides an integrated information source for our internal process users or external customers”

• “Now we have to architect the information related to business processes differently to search across multiple repositories”

• But they lack tools and applications that support this


Current solutions

• Enterprise search, portals, knowledge management and content management systems lashed up in ad hoc architectures

• Doesn’t unify data and content

• Doesn’t provide context or scope

• Too many results (requires searching the answer to the original search)


Why semantic search?

• Explicitly represented knowledge can• Unify access to both content and data

• Create context and frames of reference

• Intellectual contributions that inform the search process must be captured

• The answer should include the question


Semantic search – some definitions

• Search: the process of retrieving objects matching a given query

• Semantic search: • Search that uses an explicit

representation of knowledge to retrieve, organize or display objects matching a query

• Search that transparently renders human insight into the nature of matches


Benefits in the enterprise

• Addresses pervasive frustration with enterprise search

• Let users • Find high-value information quickly

• Add more value to it, and

• Share it with others

• Aligns information to business needs


Roots

• Parametric search

• Query by example

• Retrieval by reformulation• Rabbit, Argon

• Work in existing enterprise search and knowledge management

• Autonomy, Semio


Semantic search requires metadata• Ontologies

• Specifications of how to represent classes, instances and their properties

• Sometimes called “vocabularies”

• Controlled vocabularies• Terms for saying what something is about• Also called “taxonomies” and “thesauri”

• Instances• Descriptions of resources

• Application profiles• Specifications of which classes and properties are useful

and how they are to be used in an application


Current metadata solutions are costly

• Much custom development done• Not easy to tag or incorporate content into

the desired structures• No easy way for groups creating the

vocabularies to deliver them to production environments

• Perceived lack of tools• Point solutions not well integrated• Existing platform solutions closed


Metadata in today’s enterprises

• From thirty interviews conducted with Fortune 1000 organizations during Fall 2004

• Use of metadata not yet widespread but emerging

• Understanding varies widely across enterprises

• Three basic approaches• Top down, bottom up, and give up


Approach: top down

• CEO says “We must be an information-driven company”

• “Corporate controlled vocabulary that all divisions will use”

• Typically based on Dublin Core• Used for subject tagging

• The effort is multi-year, ROI hard to track, and may not be implemented or adopted widely


Approach: bottom up

• Groups determine their vocabulary while describing their process

• Often in a collaboration environment

• Light tagging of content when it is created or when the content is published to a portal

• Again, based on Dublin Core and their own controlled vocabularies


Approach: give up

• Assumption: too difficult to create metadata from existing content

• “We can’t ever hope to organize this morass of content, so let’s put in a search appliance like Google”

• “Our internal needs are like the public internet and users are familiar with Google searches”

• But still feel that metadata would improve matters, particularly within business units


Don’t give up!

• RDF can make metadata use easier and less costly

• An open standard for metadata reduces cost and avoids technology and vendor lock-in

• A “universal solvent” for data and content

• A platform for reuse and sharing


Building semantic search systems with RDF

• Define/reuse ontologies expressed in RDF(S)• Classes for defining instances and controlled

vocabularies• Properties for facets and additional attributes

• Import/transform instances into an RDF representation

• Resources referred to via URIs• Content and controlled vocabularies

• Write application profiles in terms of RDF


Types of semantic search in RDF

• Searching for RDF• Swoogle

• Adding value to search using RDF• TAP, FOAFNaut

• Searching resources using RDF• Edutella, Seamark


Swoogle: Searching for RDF

•Crawling for SW documents

•Leverages Google indexing

•And structure of key document types

•Searching for ontologies and instance data

•Mostly relevant to people bulding semantic applications rather than general users


TAP: Adding value to search using RDF

• Layering “related items” on top of traditional Web search

•Arm’s length integration and value-add for traditional Web search


FOAFNaut: Adding value to search using RDF

• Specialized search and visualization over FOAF networks

•Introducing the notion of social aspects of finding information


Edutella: Searching resources using RDF

•P2P architecture federating collections of learning objects•Work on distributing RDF queries using schema information•RDF as a more natural representation for learning objects than IEEE LOM


Seamark: Searching resources using RDF

•Using ontologies and taxonomies to define navigation over specific collections

•First implementation of faceted navigation using RDF


Faceted navigation as a type of semantic search

• Metadata may be faceted, i.e., includes properties whose ranges form a near-orthogonal set of controlled vocabularies

• Creator: Dickens, Charles• Subject: Arsenic, Antimony• Location: World > U.S. > California > Venice

• Facets form a frame of reference for information overview, access and discovery

• Other properties serve as landmarks and cues


Case study: DC 2003 Online Proceedings

•Further the goals of the Dublin Core Metadata Initiative (DCMI) by providing DC-centric faceted navigation of online proceedings


Project timeline• July 2003

• Initial experiment using DC 2002 site• August 2003

• Initial proposal to DCMI• Iterative prototyping involving

• Selection and development of ontologies• Generation of instance metadata• Specification of application profile

• Conversion of DC2003 dataset into navigable RDF• Elapsed time to implement: 1 day

• September 2003• Design and editing of controlled vocabulary• Final iterations on site pages• Launch at conference


Ontology

• Reused ontologies and metadata vocabularies

• Papers and posters: Dublin Core

• Creators: Friend Of A Friend (FOAF)

• Subjects: Thesaurus Interchange Format (TIF)

• Added relatively few properties and classes in a conference ontology

• Events

• Tracks


Ontology for conferences<s:Class rdf:about="&dcconf;Event">

<s:label>Presentation</s:label> </s:Class> <s:Class rdf:about="&dcconf;Paper">

<s:label>Paper</s:label> <s:subClassOf rdf:resource="&dcconf;Event"/>

</s:Class> <s:Class rdf:about="&dcconf;Track">

<s:label>Conference Track</s:label> </s:Class> <rdf:Property rdf:about="&dcconf;track">

<s:label>Track</s:label> <s:comment>The track that the given paper is in.</s:comment> <s:domain rdf:resource="&dcconf;Event" /> <s:range rdf:resource="&dcconf;Track" />

</rdf:Property>


Controlled vocabulary

• Author-assigned keywords used as source materials

• Combined author-assigned with editorial judgment about the CV terms and structure


Seed thesaurus


Wrapping author-assigned keywords<tif:Term rdf:about="&dcconf2003;Relational_Database">

<tif:value>Relational Database</tif:value> <tifs:USE rdf:resource="&dcconf2003;Relational_Databases" />

</tif:Term> <tif:Term rdf:about="&dcconf2003;relationship_metadata">

<tif:value>Relationship metadata</tif:value> <tifs:BT rdf:resource="&dcconf2003;Domain_Metadata" />

</tif:Term> <tif:Term rdf:about="&dcconf2003;requirements">

<tif:value>Requirements</tif:value> </tif:Term> <tif:Term rdf:about="&dcconf2003;resource_discovery">

<tif:value>Resource discovery</tif:value> <tifs:BT rdf:resource="&dcconf2003;Discovery" />

</tif:Term> <tif:Term rdf:about="&dcconf2003;resource-level_metadata">

<tif:value>Resource-level metadata</tif:value> <tifs:BT rdf:resource="&dcconf2003;Domain_Metadata" />

</tif:Term> <tif:Term rdf:about="&dcconf2003;SCORM">

<tif:value>SCORM</tif:value> <tifs:USE rdf:resource="&dcconf2003;Sharable_Content_Object_Reference_Model_SCORM" />

</tif:Term>


Adding editorial control<tif:Term rdf:about="&dcconf2003;Domain_Metadata">

<tif:value>Domain Metadata</tif:value> <tifs:BT rdf:resource="&dcconf2003;Applications" />

</tif:Term> <tif:Term rdf:about="&dcconf2003;Governments">

<tif:value>Governments</tif:value> <tifs:BT rdf:resource="&dcconf2003;Organizations_and_Domains" />

</tif:Term> <tif:Term rdf:about="&dcconf2003;Federal_Geographic_Data_Committee_Metadata">

<tif:value>Federal Geographic Data Committee Metadata</tif:value> <tifs:BT rdf:resource="&dcconf2003;Domain_Metadata" /> <tifs:RT rdf:resource="&dcconf2003;Governments" />

</tif:Term> <tif:Term rdf:about="&dcconf2003;Geospatial_Metadata">

<tif:value>Geospatial Metadata</tif:value> <tifs:BT rdf:resource="&dcconf2003;Domain_Metadata" /> <tifs:RT rdf:resource="&dcconf2003;Organizations_and_Domains" />

</tif:Term> <tif:Term rdf:about="&dcconf2003;Government_Agency_Metadata">

<tif:value>Government Agency Metadata</tif:value> <tifs:BT rdf:resource="&dcconf2003;Domain_Metadata" /> <tifs:RT rdf:resource="&dcconf2003;Governments" />

</tif:Term>


Instance metadata• Paper and poster metadata automatically

extracted from author submissions• Ad hoc Perl script• Manual review and cleanup of generated RDF• Mostly Dublin Core with some application-

specific properties

• Creator and organization metadata manually collated from paper and poster metadata

• Represented in FOAF (but not in the manner in which FOAF is typically used)


Papers and posters<dcconf:Paper rdf:about="http://www.siderean.com/dc2003/103_paper-22.pdf">

<seamark:texturl>http://www.siderean.com/dc2003/103_paper-22.pdf</seamark:texturl> <rdf:type rdf:resource="&dcconf;Event"/> <dcconf:track rdf:resource="&dcconf;Interoperability" /> <dc:title>Two Paths to Interoperable Metadata</dc:title> <dc:creator rdf:resource="&dcconf;Godby_Carol" /> <dc:creator rdf:resource="&dcconf;Smith_Devon" /> <dc:creator rdf:resource="&dcconf;Childress_Eric" /> <dc:description> This paper describes a prototype for a Web service that translates between pairs of metadata schemas. Despite a current trend toward encoding in XML and XSLT, we present arguments for a design that features a more distinct separation of syntax from semantics. The result is a system that auomates routine processes, has a well-defined place for human input, and achieves a clean separation of the document data model, the document translations, and the machinery of the application. </dc:description> <dc:subject rdf:resource="&dcconf2003;metadata_schema_translation" />

<dcconf:authorKeyword rdf:resource="&dcconf2003;metadata_schema_translation" /> <dc:subject rdf:resource="&dcconf2003;Web_services" />

<dcconf:authorKeyword rdf:resource="&dcconf2003;Web_services" /> <dc:subject rdf:resource="&dcconf2003;communities_of_practice" /> <dcconf:authorKeyword rdf:resource="&dcconf2003;communities_of_practice" />

</dcconf:Paper>


Creators and organizations<foaf:Person rdf:about="&dcconf;Greenberg_Jane">

<foaf:name>Greenberg, Jane</foaf:name> <foaf:mbox rdf:resource="mailto:[email protected]" /> <foaf:memberOf rdf:resource="&dcconf;University_of_North_Carolina_at_Chapel_Hill" /> <foaf:publication rdf:resource="http://www.siderean.com/dc2003/202_Paper82-color-NEW.pdf" />

</foaf:Person> <foaf:Organization

rdf:about="&dcconf;University_of_North_Carolina_at_Chapel_Hill"> <foaf:name>University of North Carolina at Chapel Hill, USA</foaf:name> <foaf:member rdf:resource="&dcconf;Greenberg_Jane" /> <foaf:member rdf:resource="&dcconf;Crystal_Abe" />

</foaf:Organization>


Application profile

• Expressed in XRBR (XML For Retrieval By Reformulation)

• Specifies a view over (possibly heterogeneous) RDF schemas with hints as to its interpretation and use for faceted navigation

• Provides a language for query reformulation and refinement in the context of navigation

• Query: “give me all resources where…” + advice

• Response: result set + suggested query refinements + original query


Application profile: specifying facets

<xrbr:query xmlns:xrbr="http://www.siderean.com/2001/10/xrbr/" item-type="http://www.dcmi.org/dcconf/objects#Event" sort-dimension="title" > <xrbr:hint flattenresults="yes" startpagecolumns="4"/> <xrbr:dimensions>

<xrbr:dimension name="title" predicate="http://purl.org/dc/elements/1.1/title"> <xrbr:hint textsearch="yes" label="Title"

function="itemlabel"/> <xrbr:return />

</xrbr:dimension> <xrbr:dimension name="description"

predicate="http://purl.org/dc/elements/1.1/description"> <xrbr:hint textsearch="yes" label="Description"

function="itemdescription"/> <xrbr:return />

</xrbr:dimension>

…</xrbr:dimensions>

</xrbr:query>


Application profile: specifying hierarchical facets

…<xrbr:dimension name="BT1" predicate="http://purl.org/dc/elements/1.1/subject"

display-predicate="http://www.w3c.rl.ac.uk/2003/07/31-tif#value" root-resource="http://www.dcmi.org/dcconf/

2003#Organizations_and_Domains" ancestor-predicate="http://www.w3c.rl.ac.uk/2003/07/31-tif-

simple#BT" > <xrbr:hint label="Organizations and Domains"

facet="yes“ scopenote="Sectors, languages, special literatures or

communities that use metadata" /> <xrbr:suggestions count="7" />

</xrbr:dimension>

…


Application profile: flattening graphs…

<xrbr:structure name="creator" predicate="http://purl.org/dc/elements/1.1/creator">

<xrbr:dimension name="creatorname" predicate="http://xmlns.com/foaf/0.1/#name"> <xrbr:hint label="Author" textsearch="yes"/> <xrbr:suggestions count="7" /> <xrbr:return />

</xrbr:dimension> <xrbr:dimension name="creatororg“

predicate="http://xmlns.com/foaf/0.1/#memberOf" display-predicate="http://xmlns.com/foaf/0.1/#name"> <xrbr:hint label="Author Affiliation" /> <xrbr:suggestions count="7" /> <xrbr:return />

</xrbr:dimension> </xrbr:structure>

…


Automatically generated interface


Alternate view: creators


Alternate view: subjects


Site start page


Site drilldown


Case study: Environmental Health News

• Aggregating news stories from the Web

• Semi-automated metadata creation by a team of subject matter experts and editors

• Semantic search to design custom feeds


Case study: Gateway to Educational Materials

•Aggregating learning objects from members of the GEM Consortium

•Embedding semantic search into a portal


Case study: NASA JPL

•Project information aggregated from content and data repositories

•Using and extending taxonomies

•Exploiting document type/genre


Related work in RDF• OCLC

• Metadata Switch

• MIT• Simile

• Longwell• Haystack

• Aduna• Sesame

• Ontoprise• OntoSeek

• Nature Publishing Group• Urchin


Issues• Scale: must be commensurate with expectations and

requirements from traditional web and enterprise search• Number of objects, feeds: 106 to 109

• Ingest rates: ~ 103 – 104 triples/sec, how many per resource?• Tagging: where and when?• Latency: < 0.5 sec user time regardless of application

• Retrieval algorithms: many alternatives still being explored• Federated services vs. centralized servers• Relationship to relevance ranking• Support for aggregate and text search operators in RDF query

• Usability: lots of work to be done to validate benefits• Navigation• Precision and recall• Visualization

• Security, trust and provenance: just beginning to understand


Lessons

• Balanced incremental approach

• Leverage metadata and indices at hand

• Exploit statistics where desirable• But layer a framework on top to structure

the statistics

• Significant mileage from very simple frameworks


Lessons: ontologies• Don’t do: assume you have to build elaborate

OWL ontologies • Don’t have to boil the ocean to get the benefits• OWL DL, are OWL Full are overkill for this class

of application

• Do: Tiny Ontologies Stitched Together• RDF Schema with a smattering of RDF/OWL

properties (e.g., owl:inverse)• Start with DC + SKOS + FOAF


Lessons: controlled vocabularies

• Don’t do: huge monolithic taxonomies• Unless they are ready at hand and can be

reused largely without modification

• Do: bite-sized controlled vocabularies that exploit faceted approaches

• 4 facets x 10 terms per facet versus 104

terms in a single taxonomy• Start with flat term lists• Add BT/NT/RT relationships over time


Lessons: instances

• Manual creation• Don’t do: exhaustive author creation of metadata• Do: community annotation and tagging

• (Semi-)automated creation• Don’t do: assume elaborate information

extraction based on NLP, subject tagging and categorization

• Do: quick and dirty NEE or better yet, stick to readily available asset and relational metadata (date, creator, document type/genre)

• Much of the benefit at a fraction of the effort


Application profiles

• Metadata is increasingly pervasive• The way to leverage existing information

infrastructure

• Exploit “on-demand” information integration feature of RDF

• DB + XML -> XLST - > RDF(S)


The big question: statistics vs. knowledge

• Statistics can’t deliver everything• Alan Kay’s puppy analogy• Vitanyi work on “Google learning”

• On the other hand, knowledge is dearly won• CYC

• Need a balance that enables adoption without losing the benefits

• Lessons from• Statistics vs. knowledge in NLP• Expert systems


Future directions

• User tagging + RDF: the killer SW application?

• The rehabilitation of metadata in the social software community

• The re-emergence of RSS/RDF

• “Folksonomy”-driven collaborative search

• Del.icio.us, Flickr, CiteULike

• Growth of the SW compared to historical growth of the Web: it’s 1994 all over again


Summary

• Semantic search has a role in today’s enterprises

• RDF provides a framework that can ease adoption and encourage innovation in semantic search

• The future for enterprise and consumer use looks bright

semantic search using rdf metadata (semtech 2005)

Technology

search process

query search

definitions search

search appliance

search capabilities

original search

semantic search systems

types of semantic search