knowledge enabled information and services science realizing the relationship web: morphing...

Knowledge Enabled Information and Services Science

Realizing the Relationship Web: Morphing information access on the Web from today’s document- and

entity-centric paradigm to a relationship-centric paradigm

ACM Multimedia International Workshop on the Many Faces of Multimedia Semantics

September 28, 2007, Augsburg, Germany

Amit ShethKno.e.sis Center, Wright State University,

Dayton, OH

This talk also represents work of several members of Kno.e.sis team, esp. theSemantic Discovery and Semantic Sensor Data. http://knoesis.wright.edu

Thanks, M. Perry, C. Ramakrishnan, C. Thomas


Objects of Interest

“An object by itself is intensely uninteresting”.

Grady Booch, Object Oriented Design with Applications, 1991

Keywords

|

Search

Entities

|

Integration

Relationships

|

Analysis,Insight

Entities + Relationships also needed to model & study Events


Semantics and Relationships

Increasing depth and sophistication in dealing with semantics by dealing with (identifying/searching to analyzing) documents, entities, and relationships.

Documents/Media

Entities

RelationshipsFuture

Current

Past


Data, Information, and InsightData, Information, and Insight

What: Which thing or which particular one Who: What or which person or persons Where: At or in what place When: At what time

How: In what manner or way; by what means Why: For what purpose, reason, or cause; with

what intention, justification, or motive

© Ramesh Jain


Insights require understanding RelationshipsInsights require understanding Relationships

Object Location Time Relationships

What X X X

Who X

Where X

When X

How X

Why X

© Ramesh Jain


Semantics and Relationships

Semantics is derived from relationships. Consider the linguistics perspective.

“Semantics is the study of meaning. …We may distinguish a number of legitimate ways to approach semantics:

• …• the relationship between linguistic expressions (e.g.

synonymy, antonymy, hyperonymy, etc.): sense; • the relationship to linguistic expressions to the "real

world": reference. “

Ontologies use KR language to support modeling of relationships.Quoted part from http://www.ncl.ac.uk/sml/staff/. © 2000 Jonathan West.


Why is this a hard problem?

Are objects/entities equivalent/equal(same)?How (well) are they related? • Implicit vs explicit; formal/assertional vs social

consensus based; powerful (beyond FOL): partial, probilistic and fuzzy match

• Degrees of relatedness and relevance: semantic similarity, semantic proximity, semantic distance, ….– [differentiation, disjointedness]– related in a “context”

• Even is-a link involves different notions: identify, unity, essense (Guarino and Wetley 2002)

Semantic ambiguity, also based on incomplete, inconsistent, approximate information/knowledge


Issues - Relationships

• Identifying Relationship (extraction)• Expressing (specifying, representing)

relationships • Discovering and Exploring Relationships• Hypothesizing and Validating Relationships• Utilizing/exploiting Relationships for

Semantic Applications (in document search, querying metadata, inferencing, analysis, insight, discovery)


Information Extraction for Metadata Creation

WWW, EnterpriseRepositories

METADATAMETADATA

EXTRACTORSEXTRACTORS

Digital Maps

NexisUPIAPFeeds/

Documents

Digital Audios

Data Stores

Digital Videos

Digital Images. . .

. . . . . .

Create/extract as much (semantics)metadata automatically as possible;Use ontlogies to improve and enhanceextraction


Automatic Semantic Metadata Extraction/Annotation


Limited tagging(mostly syntactic)

COMTEX Tagging

Content‘Enhancement’Rich Semantic

Metatagging

Value-added Voquette Semantic Tagging

Value-addedrelevant metatagsadded by Voquetteto existing COMTEX tags:

• Private companies • Type of company• Industry affiliation• Sector• Exchange• Company Execs• Competitors

Semantic Annotation (Extraction + Enhancement)


Enabling powerful linking of actionable information and facilitating important semantic applications such as knowledge discovery and link analysis

(user’s task of manually retrieving all the information he needs to know is greatly minimized; he can spend more time making effective decisions)

Semantic Metadata Content TagsCompany: Cisco Systems, Inc.Classification: Channel Partners,

E-Business SolutionsChannel Partner: Siemens NetworkChannel Partner: Voyager NetworkChannel Partner: Siemens NetworkChannel Partner: Wipro GroupE-Business Solution: CI S-1270 SecurityE-Business Solution: CI S-320 LearningE-Business Solution: CI S-6250 FinanceE-Business Solution: CI S-1005 e-MarketTicker: CSCOI ndustry: Telecommunication, . . .Sector: Computer HardwareExecutive: J ohn ChambersCompetition: Nortel Networks

Syntactic MetadataProducer: BusinessWireSource: BloombergDate: Sept. 10 2001Location: San J ose, CAURL: http:/ /bloomberg.com/1.htmMedia: Text

XML content item with enriched semantic tagging, ready to be queried

E-Business SolutionOntology

CiscoSystems

VoyagerNetwork

SiemensNetwork

WiproGroup

UlysysGroup

CIS-1270 Security

CIS-320Learning

CIS-6250 Finance

CIS-1005 e-Market

Channel Partner

belongs to

- - -

Ticker

represen

ted b

y

- - -

- - -

- - -

- - -

Industry

chan

nel p

artn

er of

- - -

- - -

- - -

- - -

Competitioncompetes with

provider of

- - -

- - -

- - -

- - -

Executives

works

for

- - -

- - -

- - -

- - -

Sectorbelo

ngs

to

Semantic Enhancement

Uniquelyexploiting

real-worldsemantic

associationsin the right

context

SemanticMetadataExtraction

(also syntactic)

Content TagsSemantic MetadataClassification: Channel Partners,

E-Business SolutionsCompany: Cisco Systems, Inc.

Syntactic MetadataProducer: BusinessWireSource: BloombergDate: Sept. 10 2001Location: San J ose, CAURL: http: //bloomberg.com/1.htmMedia: Text

ChannelPartners

E-BusinessSolutionsClassification

Content Tags

Semantic MetadataClassification: Channel Partners,

E-Business Solutions

Classification CommitteeKnowledge-base, Machine Learning &

Statistical Techniques

Semantic Metadata Enhancement


Video withEditorialized Text on the Web

AutoCategorization

AutoCategorization

Semantic MetadataSemantic Metadata

Automatic Classification & Metadata Extraction (Web page)


Extraction Agent

Enhanced Metadata Asset

Ontology-directed Metadata Extraction (Semi-structured data)

Web Page


830.9570 194.9604 2

580.2985 0.3592

688.3214 0.2526

779.4759 38.4939

784.3607 21.7736

1543.7476 1.3822

1544.7595 2.9977

1562.8113 37.4790

1660.7776 476.5043

parent ion m/z

fragment ion m/z

ms/ms peaklist data

fragment ionabundance

parent ionabundance

parent ion charge

ProPreO: Ontology-mediated provenance

Mass Spectrometry (MS) Data

Semantic Extraction/Annotation of Experimental Data


Video metadata and search

Technique Who’s trying it How it works Wired Tired

Scanning the script

Blinkx, TVEyes Everything said in a clip is tracked through voice recognition software, closed-caption information, or a combination of the two.

Hunts down TV news references to, say, Lindsay Lohan.

A mere mention of her name doesn't guarantee Lindsay is in the clip.

Identifying what's being shown

Google, UCSD's Statistical Visual Computing Lab, VideoMining

Algorithms try to figure out what's in the video by monitoring attributes like behavior and movement (VideoMining), faces (Google), and objects (UCSD).

Finds friends, public figures, or specific actions like car chases.

The tech is stuck in the lab or limited to specialized search tasks.

Analyzing links and metadata

Dabble, Google Conventional search spiders scan the text around a video and the pages that link to it.

The fastest and best way to search for video data.

Can't "see" what's in the videos or locate specific action in a long clip.

Wired Mag, Aug 2007

Knowledge Enabled Information and Services Science 18

Semantic Sensor ML – Adding Ontological Metadata

Person

Company

Coordinates

Coordinate System

Time Units

Timezone

SpatialOntology

DomainOntology

TemporalOntology

Mike Botts, "SensorML and Sensor Web Enablement," Earth System Science Center, UAB Huntsville


Relationships on the Web:early work


MREF (Metadata Reference Link -- complementing HREF)

Creating “logical web” through

Media Independent Metadata based Correlation


Metadata Reference Link (<A MREF …>)

<A HREF=“URL”>Document Description</A>

physical link between document (components)

• <A MREF KEYWORDS=<list-of-keywords>; THRESH=<real>>Document Description</A>

• <A MREF ATTRIBUTES(<list-of-attribute-value-pairs>)>Document Description</A>


Creating “logical web” through Media Independent Metadata based Correlation

MREFMetadata Reference Link -- complementing HREF (1996, 1998)

METADATA

DATA

METADATA

DATAMREFin RDF

ONTOLOGYNAMESPACE

ONTOLOGYNAMESPACE


Model for LogicalCorrelation using Ontological Terms and Metadata

Framework for Representing MREFs

Serialization(one implementation choice)

MREF (1998)

K. Shah and A. Sheth, "Logical Information Modeling of Web-accessible Heterogeneous Digital Assets", Proc. of the Forum on Research and Technology Advances in Digital Libraries," (ADL'98), Santa Barbara, CA, May 28-30, 1998, pp. 266-275.


height, width and size

water.gif (Data)Metadata Storage

water.gif

……mpeg

……ppm

Major component(RGB)Major component(RGB)

Blue

Content based MetadataContent based Metadata

ContentDependentMetadata

Correlation based on Content-based Metadata

Some interesting information on dams is available here.“information on dams” is defined by an MREF defining keywords and metadata (which may be used for a query).


Domain Specific Correlation

Potential locations for a future shopping mall identified by all regions having a population greater than 500 and area greater than 50 sq meters having an urban land cover and moderate relief <A MREF ATTRIBUTES(population < 500; area < 50 & region-type = ‘block’ & land-cover = ‘urban’ & relief = ‘moderate’)>can be viewed here</A>

=> media-independent relationships between domain specific metadata: population, area, land cover, relief

=> correlation between image and structured data at a higher domain specific level as opposed to physical “link-chasing” in the WWW


TIGER/Line DB

Population: Area:

Boundaries:

Land cover:Relief:

Census DB

Map DB

Regions(SQL)

Boundaries

Image Features(IP routines)

Repositories and the Media Types


Complex Relationships

• Some relationships may not be manually asserted, but according to statistical analyses of text, experimental data, etc.

allow association of provenance data with classes, instances, relationship types and direct relationships or statements


A simple relationship ?

Smoking CancerCauses


Complex Relationships - Cause-Effects & Knowledge discovery

VOLCANO

LOCATIONASH RAIN

PYROCLASTICFLOW

ENVIRON.

LOCATION

PEOPLE

WEATHER

PLANT

BUILDING

DESTROYS

COOLS TEMP

DESTROYS

KILLS


Knowledge Discovery - Example

Earthquake Sources Nuclear Test Sources

Nuclear Test May Cause Earthquakes

Is it really true?

Complex Relationshi

p:How do you model

this?


Inter-Ontological Relationships

A nuclear test could have caused an earthquakeif the earthquake occurred some time after thenuclear test was conducted and in a nearby region.

NuclearTest Causes Earthquake

<= dateDifference( NuclearTest.eventDate,

Earthquake.eventDate ) < 30

AND distance( NuclearTest.latitude,

NuclearTest.longitude,

Earthquake,latitude,

Earthquake.longitude ) < 10000


Knowledge Discovery – exploring relationship…

For each group of earthquakes with magnitudes in the ranges5.8-6, 6-7, 7-8, 8-9, and >9 on the Richter scale per yearstarting from 1900, find number of earthquakes

Number of earthquakes with magnitude > 7 almost constant. So nuclear tests probably only cause earthquakes with magnitude < 7


Now possible – Extracting relationships between MeSH terms from PubMed

Biologically active substance

LipidDisease or Syndrome

affects

causes

affectscauses

complicates

Fish Oils Raynaud’s Disease???????

instance_of instance_of

UMLS Semantic Network

MeSH

PubMed9284 documents

4733 documents

5 documents


Schema-Driven Extraction of Relationships from Biomedical Text

Cartic Ramakrishnan, Krys Kochut, Amit P. Sheth: A Framework for Schema-Driven Relationship Discovery from Unstructured Text. International Semantic Web Conference 2006: 583-596 [.pdf]


Method – Parse Sentences in PubMed

SS-Tagger (University of Tokyo)

SS-Parser (University of Tokyo)

(TOP (S (NP (NP (DT An) (JJ excessive) (ADJP (JJ endogenous) (CC or) (JJ exogenous) ) (NN stimulation) ) (PP (IN by) (NP (NN estrogen) ) ) ) (VP (VBZ induces) (NP (NP (JJ adenomatous) (NN hyperplasia) ) (PP (IN of) (NP (DT the) (NN endometrium) ) ) ) ) ) )

• Entities (MeSH terms) in sentences occur in modified forms• “adenomatous” modifies “hyperplasia”• “An excessive endogenous or exogenous stimulation” modifies “estrogen”

• Entities can also occur as composites of 2 or more other entities• “adenomatous hyperplasia” and “endometrium” occur as “adenomatous hyperplasia of the endometrium”


Method – Identify entities and Relationships

in Parse TreeTOP

NP

VP

S

NPVBZ

induces

NPPP

NPINof

DTthe

NNendometrium

JJadenomatous

NNhyperplasia

NP PP

INby

NNestrogenDT

the

JJexcessive ADJP NN

stimulation

JJendogenous

JJexogenous

CCor

MeSHIDD004967MeSHIDD006965 MeSHIDD004717

UMLS ID

T147

ModifiersModified entitiesComposite Entities


Resulting Semantic Web Data in RDF

ModifiersModified entitiesComposite Entities

estrogen

An excessive endogenous or

exogenous stimulation

modified_entity1composite_entity1

modified_entity2

adenomatous hyperplasia

endometrium

hasModifier

hasPart

induces

hasPart

hasPart

hasModifier

hasPart


Blazing Semantic Trails in Biomedical Literature

Cartic Ramakrishnan, Amit P. Sheth: Blazing Semantic Trails in Text: Extracting Complex Relationships from Biomedical Literature. Tech. Report #TR-RS2007 [.pdf]


Relationships -- Blazing the Trails

“The physician, puzzled by her patient's reactions, strikes the trail established in studying an earlier similar case, and runs rapidly through analogous case histories, with side references to the classics for the pertinent anatomy and histology. The chemist, struggling with the synthesis of an organic compound, has all the chemical literature before him in his laboratory, with trails following the analogies of compounds, and side trails to their physical and chemical behavior.” [V. Bush, As We May Think. The Atlantic Monthly, 1945. 176(1): p. 101-108. ]


Original documents

PMID-10037099

PMID-15886201


Semantic Trail


Semantic Trails over all types of Data

Semantic Trails can be built over a Web of Semantic (Meta)Data

extracted (manually, semi-automatically and automatically) and gleaned from

• Structured data (e.g., NCBI databases)

• Semi-structured data (e.g., XML based and semantic metadata standards for domain specific data representations and exchanges)

• Unstructured data (e.g., Pubmed and other biomedical literature)

and• Various modalities (experimental data, medical images, etc.)


ApplicationsApplications

“Everything's connected, all along the line. Cause and effect.

That's the beauty of it.

Our job is to trace the connections and reveal them.”

Jack in Terry Gilliam’s 1985 film - “Brazil”


Watch list Organization

Company

Hamas

WorldCom

FBI Watchlist

Ahmed Yaseer

appears on Watchlist

member of organization

works for Company

Ahmed Yaseer:

• Appears on Watchlist ‘FBI’

• Works for Company ‘WorldCom’

• Member of organization ‘Hamas’

An application in Risk & Compliance


Global Investment Bank

Example of Fraud

prevention application used in financial services

User will be able to navigate the ontology using a number of different interfaces

World Wide Web content

Public Records

BLOGS,RSS

Un-structure text, Semi-structured Data

Watch ListsLaw

Enforcement Regulators

Semi-structured Government Data

Scores the entity based on the content and entity relationships

EstablishingNew Account


Hypothesis driven retrieval of Scientific Text


Semantic Browser


More about the Relationship Web

Relationship Web takes you away from “which document” could have information I need, to “what’s in the resources” that gives me the insight and knowledge I need for decision making.

Amit P. Sheth, Cartic Ramakrishnan: Relationship Web: Blazing Semantic Trails between Web Resources. IEEE Internet Computing July 2007 (to appear) [.pdf]


Events: 3 Dimensions – Spatial, Temporal and Thematic

Spatial

Temporal

Thematic


Events and STT dimensions

• Powerful mechanism to integrate content– Describes the Real-World occurrences– Can have video, images, text, audio all of the same event– Search and Index based on events and STT relations

• Many relationship types– Spatial:

• What events happened near this event? • What entities/organizations are located nearby?

– Temporal: • What events happened before/after/during this event?

– Thematic:• What is happening?• Who is involved?

• Going further– Can we use What? Where? When? Who? to answer Why? / How?– Use integrated STT analysis to explore cause and effect


Example Scenario: Sensor Data Fusion and Analysis

High-level Sensor (S-H)

Low-level Sensor (S-L)

A-H E-H A-L E-L

H L

• How do we determine if A-H = A-L? (Same time? Same place?)

• How do we determine if E-H = E-L? (Same entity?)

• How do we determine if E-H or E-L constitutes a threat?


Sensor Data Pyramid

Raw Sensor (Phenomenological) Data

Feature Metadata

Entity Metadata

Relationship

Metadata

Expr

essi

vene

ss

Data

Information

Semantics/Understanding/Insight

Data Pyramid


Collection

Feature Extraction

Entity Detection

RDFKB

Semantic Analysis

Data

• Raw Phenomenological Data

TML

Fusion

SML-S

SML-S

SML-S

Ontologies

Sensor Data Architecture

Information

• Entity Metadata

• Feature Metadata

Knowledge

• Object-Event Relations

• Spatiotemporal Associations

• Provenance Pathways

Sensors (RF, EO, IR, HIS, acoustic)

• Object-Event Ontology

• Space-Time Ontology

Analysis Processes Annotation Processes

O&M


Current Research Towards STT Relationship Analysis

• Modeling Spatial and Temporal data using SW standards (RDF(S))1

– Upper-level ontology integrating thematic and spatial dimensions– Use Temporal RDF3 to encode temporal properties of relationships– Demonstrate expressiveness with various query operators built

upon thematic contexts

• Graph Pattern queries over spatial and temporal RDF data2

– Extended ORDBMS to store and query spatial and temporal RDF– User-defined functions for graph pattern queries involving spatial

variables and spatial and temporal predicates– Implementation of temporal RDFS inferencing

1. Matthew Perry, Farshad Hakimpour, Amit Sheth. "Analyzing Theme, Space and Time: An Ontology-based Approach", Fourteenth International Symposium on Advances in Geographic Information Systems (ACM-GIS '06), Arlington, VA, November 10 - 11, 2006

2. Matthew Perry, Amit Sheth, Farshad Hakimpour, Prateek Jain. “Supporting Complex Thematic, Spatial and Temporal Queries over Semantic Web Data", Second International Conference on Geospatial Semantics (GeoS ‘07), Mexico City, MX, November 29 – 30, 2007

3. Claudio Gutiérrez, Carlos A. Hurtado, Alejandro A. Vaisman. “Temporal RDF”, ESWC 2005: 93-107


Upper-level Ontology modeling Theme and Space

OccurrentContinuant

Named_PlaceSpatial_OccurrentDynamic_Entity

Spatial_Region

Occurrent: Events – happen and then don’t existContinuant: Concrete and Abstract Entities – persist over timeNamed_Place: Those entities with static spatial behavior (e.g. building)Dynamic_Entity: Those entities with dynamic spatial behavior (e.g. person)Spatial_Occurrent: Events with concrete spatial locations (e.g. a speech)Spatial_Region: Records exact spatial location (geometry objects,

coordinate system info)

occurred_at located_at

occurred_at: Links Spatial_Occurents to their geographic locationslocated_at: Links Named_Places to their geographic locations

rdfs:subClassOfproperty


OccurrentContinuant

Named_Place

Spatial_OccurrentDynamic_Entity

PersonCity

Politician

Soldier

Military_Unit

BattleVehicle

Bombing

Speech

Military_Eventassigned_to

on_crew_of

used_in

gives

participates_in

trains_at

Spatial_Region

located_at occurred_at

Upper-level Ontology

Domain Ontology

rdfs:subClassOf used for integrationrdfs:subClassOfrelationship type


Temporal RDF Graph: Platoon Membership

E1:Soldier

E3:Platoon E5:Soldier

E2:Platoon

E4:Soldier

assigned_to [1, 10]

assigned_to [11, 20]

assigned_to [5, 15]

assigned_to [5, 15]

E1 is assigned to E2 from time 1 to 10 and then assigned to E3 from time 11 to 20

Also need to handle inferencing:

(x rdf:type Grad_Student):[2004, 2006] AND(x rdf:type Undergrad_Student):[2000, 2004] (x rdf:type Student):[2000, 2006]

Time interval represents valid time of the relationship


ORDBMS Implementation: DB Structures

Unlike thematic relationships which are explicitly stated in the RDF graph, many spatial and temporal relationships (e.g., distance) are implicit and require additional computation


Sample STT Query

Scenario (Biochemical Threat Detection): Analysts must examine soldiers’ symptoms to detect possible biochemical attack

Query specifies

(1)a relationship between a soldier, a chemical agent and a battle location (graph pattern 1)

(2)a relationship between members of an enemy organization and their known locations (graph pattern 2)

(3)a spatial filtering condition based on the proximity of the soldier and the enemy group in this context (spatial Constraint)


The Machine Factor

Formal representation of knowledge

– RDF(S), OWL, etc.Statistical analysis

– Similarity– Cooccurrence– Clustering

Intelligent aggregation of knowledge

– Collaboration/Problem Solving Environments– Decision support tools


Putting the man back in Semantics

The Semantic Web focuses on artificial agents“Web 2.0 is made of people” (Ross Mayfield)“Web 2.0 is about systems that harness collective

intelligence.” (Tim O’Reilly)The relationship web combines the skills of humans

and machines


Putting the man back in Semantics

“Web 2.0 is made of people” (Ross Mayfield)“Web 2.0 is about systems that harness collective intelligence.”

(Tim O’Reilly)The Semantic Web focuses on artificial agentsThe relationship web combines the skills of humans and machines


Going places …

Formal

Social,Informal

Implicit

Powerful