controlled ontology evolution through semiotic-based ontology evaluation international workshop on...

Controlled Ontology Evolution Through Semiotic-based Ontology Evaluation

International Workshop on Ontology Dynamics IWOD 2008

Renata [email protected]

Daniel [email protected]

Overview

Motivation Semiotic-based Ontology Evaluation Controlled Ontology Evolution through Evaluation Semiotic-based Ontology Evaluation Tool Evaluation Results Conclusion

Motivation = Reason for changes!

not good for my application?Apply changes!

not good for my intend of

use?Apply changes!

Inconsistent?Apply changes!

not aligned to dependent ontologies?

Apply changes!

Motivation = Evaluation criterion!

Which version is the best one

for my app?

Is the ontology still good (or better) for my intend of use?

Is the ontology consistent?

Can I apply these changes

without affecting the dependent

ontologies?

Overview

Motivation Semiotic-based Ontology Evaluation

Semiotics Semiotics & Ontology Semiotics & Ontology Evaluation

Controlled Ontology Evolution through Evaluation Semiotic-based Ontology Evaluation Tool Evaluation Results Conclusion

Yojo

Concept

SymbolObject

The Meaning Triangle: (Ogden and Richards 1923)

Semiotics is composed of three fundamental

components: (Moris, 1938)

Syntax

Semantics

Pragmatics

(Niles & Pease, 2001)

Language

Syntax

Semantics

Pragmatics

OntologySemiotic Object

Semantics

Pragmatics

Syntax

SymbolObject

Concept

SymbolOntology Graph

graph-like structures containing terms and their

inter-relationships

ObjectIntended Conceptualization

represent an intended conceptualization.

ConceptCommunication Context

Rep

The ontology’s representation is interpretable by some agent

(Gangemi at al, 2006)

Structural Measures

DepthBreathModularity …

Syntax

(Topological Dimension)

ConsistencyComplexityConcept Satisfiability …

Formal Semantics

(Logical Dimension)

Semiotic Measures

Pragmatics Measures

Expert‘s judgmentsUser satisfactionAgreement satisfaction

Data SetTask AssessmentModularity Assessment…

Precision-Recall Based Measures

(Functional Dimension)

FunctionalMeasures

Assesing the ontology pragmatics

Assesing the ontology cognitive semantics

Assesing the ontology syntax and formal semantics

Annotations/Documentation:Structural-Related: Depth, Breath, Modularity.Functional-Related: Expert‘s Judgments, Data Set.User-Oriented: Deployment, Commercial , History/Review, Version.

Ontology profile (Usability Dimension)

Overview


Controlled Ontology Evolution through Evaluation

Semiotic Ontology Evaluation Process

Capturing RepresentationSemantics of

changePropagation Implementation Validation


(Stojanovic, 2004)


Semiotic Ontology Evaluation Process

Capturing

RepresentationSemantics of



Explicit RequirementsReasons for changes = Evaluation Criteria


Capturing




StructuralEvaluation


Capturing





Making the ontology changes visible in a form of an adequate representation

Ontology changes need to be managed such that the ontology remains consistent


Capturing





FunctionalEvaluation


Capturing






Ontology changes need to be managed such that the ontology remains consistent

Verify consistency of dependent ontologies


Capturing






PragmaticsEvaluation


Capturing







User is able to approve the changes applied or to reverse them


Capturing







Overview

Motivation Semiotic-based Ontology Evaluation Controlled Ontology Evolution through Evaluation Semiotic-based Ontology Evaluation Tool

Implementation

Evaluation Results Conclusion

Semiotic-based Ontology Evaluation Tool




Semiotic Measures

Consistency Checking:

onto changes remains consistent

Task-based Approach:

onto. changes max. performance

Annotation Analysis:

changes reported for versioning

Implementation:


Validation

Semantics ofchange

Propagation


change

Use of the reasoners RACER System and Pellet.

„…a logical theory is consistent if it does not contain a contradiction, or, more precisely, for no proposition φ are both φ and ¬φ provable.“

Person Seal

Disjoint (Person, Seal)

Shark (primitive class)Animal andeats some (Person and Seal)

Inconsistent

Consistency Checking

Task-based Approach

How effective a given ontology is in the light of a well-defined task (Porzel, 2004; Maedche & Staab,2002)

Task

Application

Performance Results

Ontology

Compare with Gold Standard

AnswersImprovements ?

Task-based Approach

Evaluate different ontology versions!

OntologyV0.1

OntologyV0.2

changes

Is my evolved ontology still good (or better) for my

intend of use?

max. performance for a specific task!

Task-based Approach

How efficient is the system to answers questions using just ontologies

Improvements

Question-Answering

SmartWeb

Performance Results

Compare to Gold Standard

Answers

SWIntOV0.2

V0.1

Task-based Approach

1. Plug the Evolved Ontology

4. Compare with GS Answers

2. Query the system

3. Check Time Performance

5. Make Report

• Lexicon• Taxonomy• Semantic Relations

SmartWeb

Performance Results

Compare withGold Standard

Answers

Evolved

Improvements6. Apply changes!

Usability-Related Evaluation

Annotation Analysis: Quantitative analysis of the amount of metadata linked to the tag ”rdf:comments”

<owl:AnnotationProperty rdf:ID=“structural“/><owl:AnnotationProperty rdf:ID=“functional“/><owl:AnnotationProperty rdf:ID=“user-oriented“/>

<owl:AnnotationProperty rdf:ID=“consistency“> <rdfs:subPropertyOf rdf:resource=“#structural“/></owl:AnnotationProperty><owl:AnnotationProperty rdf:ID=“task-assessment“> <rdfs:subPropertyOf rdf:resource=“#functional“/></owl:AnnotationProperty>…

<owl:Class rdf:ID="Teacher"> <rdfs:comment>Teacher Class </rdfs:comment> <rdfs:subClassOf rdf:resource="#Person"/> </owl:Class>

Overview


Evaluation Results

SWIntO Ontology (SmartWeb Project*)

Foundational (DOLCE) and general (SUMO) knowledge Domain- and task-specific knowledge

Football (soccer) entities and events Navigation Linguistic information Discourse Multimedia

* http://www.smartweb-project.org/

SmartDOLCE:Entity

SmartSUMO:Attribute

SmartSUMO:SocialRole

SportEvent:FootballPlayer SportEvent:FootballOrganizationPerson

…

… …

……

……

Consistency Checking

Functional Evaluation I

SWIntO V.0.3.2

Q1:Which matches took place in the semifinals in 1954?

Q2:Who was the world champion in 1990?

Time-performance:31,10 msGS-performance:26,23 ms

Vocabulary Overlap = 100%Hierarchy Overlap = 87%Relation Overlap = 45%

Evaluated Relation:

GS Relation:

SWIntO V.0.3.3

Q1:Which matches took place in the semifinals in 1954?

Q2:Who was the world champion in 1990?

Time-performance:31,10 msGS-performance:26,23 ms

Vocabulary Overlap = 100%Hierarchy Overlap = 87%Relation Overlap = 45%

List of Overlap Descriptions:

Evaluated Relation:

GS Relation:

Functional Evaluation II

Annotation Analysis

Conclusion

Evaluation framework to support and control ontology evolution Apply changes to an ontology keeping its quality with respect to the purpose

of the ontology (or the purpose of the ontology changes) Controlled evolution by assessing the quality of the ontology with

respect to all semiotic dimensions -> Ontology changes captured by ontology evaluation process

Implementation = choose three measures which are essential in any ontology evolution/evaluation process

Structural Dimension: Consistency Checking Functional Dimension: Task-based Evaluation Usability Dimension: Annotation Analysis

Future Work level of granularity & integration

Thank you for your attention!

RACER System, Renamed abox and concept expression reasoner . http://www.racer-system.comR. Porzel and R. Malaka, A task-based approach for ontology evaluation, 2004.M. Ciaramita J. Lehmann A. Gangemi, C. Catenacci, Modelling ontology evaluation and validation.V. Sugumaran A. Burton-Jones, V. C. Storey and P. Ahluwalia, A semiotic metrics suite for assessing the quality of

ontologies, 2004.Janez Brank, Marko Grobelnik and Dunja Mladenic, A Survey of Ontology Evaluation Techniques.W. Wahlster, Smartweb: Mobile applications of the semantic web, In Proceedings of Informatik. 2004s. P. Buitelaar and A. Frank,Ontology-driven Predicate-Argument Structure Analysis for Event AnnotationJohn F. Sowa, Ontology, Metadata, and Semiotics, 2000.P. Cimiano,Text Analysis and Ontologies, 2006.C.W. Morris, Foundations of a theory of signs. In: International Encyclopedia of Unified Science (O. Neurath, R. Carnap & C.

Morris, eds), Chicago University Press, Chicago, pp. 77-138. 1938.I. Niles and A. Pease .Towards a Standard Upper Ontology, 2001.Y. Sure and R. Studer. On-to-knowledgemethodology - final version. Technical Report Deliverable 18, Institute AIFB,

University of Karlsruhe, 2002.A. Gangemi, C. Catenacci, M. Ciaramita, and J. Lehmann. Qood grid: A metaontology-based framework for ontology

evaluation and selection. In Proceedings of EON2006, 2006.N. Guarino.. Towards a Formal Evaluation of Ontology Quality, pages 1541–1672. IEEE Intelligent Systems, 2004.D.Orbele et al. Dolce ergo sumo: On foundational and domain models in swinto (smartweb integrated ontology), 2006..N. Noy. Evaluation by ontology consumers. pages 1541–1672. IEEE Intelligent Systems, 2004.K. Dellschaft and S. Staab. On How to Perform a Gold Standard Based Evaluation of Ontology Learning, 2006A. Maedche and S. Staab.Measuring Similarity between Ontologies, 2002.L. Stojanovic. Methods and tools for ontology evolution. PhD thesis, University of Karlsruhe (TH), 2004.

References

http://www.racer-system.com/

controlled ontology evolution through semiotic-based ontology evaluation international workshop on...

Documents

evaluation stojanovic

evaluation methods

evaluation criteria

ontology syntax

semioticbased evaluation

ontology consistent

ontology changes visible

ontology pragmatics