informatics and telematics institute centre for research and technology hellas iti-certh amsterdam,...

Informatics and Telematics InstituteCentre for Research and Technology Hellas

ITI-CERTH

Amsterdam, Multimedia Semantics XG, 10-11 July 2006

Vasileios Papastathis, Research Fellow

Centre for Research and Technology Hellas (CERTH)

Informatics and Telematics Institute (ITI) (http://www.iti.gr)

Director Prof. Michael Strintzis

Multimedia Knowledge Group (http://mkg.iti.gr)Head Dr. Yiannis Kompatsiaris

Thessaloniki, GREECE


2

Presentation Overview

• ITI Multimedia Knowledge Group• Towards a Common Multimedia

Ontology Framework• IST BOEMIE Project


3

ITI Multimedia Knowledge Group

• Semantic Multimedia Analysis• Multimedia Indexing and Retrieval• Multimedia and the Semantic Web• Knowledge Structures, Languages and Tools for

Multimedia• Reasoning and Personalization for Multimedia Applications

• Ontology structures in OWL for knowledge-assisted multimedia analysis and reasoning

• Rules in SWRL including multimedia features for analysis and reasoning

• MPEG-7 Visual part representation in RDF• Graphical tool for ontology-based annotation of images

(M-Ontomat)


4

R&D Projects

• “aceMedia: Integrating knowledge, semantics and content for user centred intelligent media services”, IP 2004-2007

• “KnowledgeWeb: Realizing the Semantic Web”, funded by the DG XIII, NoE 2004-2007

• “MESH: Multimedia Semantic Syndication for Enhanced News Services”, IST – IP, 2006-2008

• “K-Space: Knowledge Space of Semantic Inference for Automatic Annotation and Retrieval of Multimedia Content, IST - NoE, 2006-2009

• “X-Media: Knowledge Sharing and Reuse Across Media”, IST – IP, 2006-2009

• “BOEMIE: Bootstrapping Ontology Evolution with Multimedia Information Extraction”, IST – STREP, 2006-2008

• “PATExpert: Advanced Patent Document Processing Techniques”, IST-STREP, 2006-2008 (collaboration with AUTH)


Towards a common Multimedia Ontology Framework

Requirements and Harmonization approaches


6

Multimedia Ontology Framework

• Framework for collecting existing multimedia knowledge representation

• Define approach for harmonization• Initiated by aceMedia in EWIMT 2005

but not directly connected to aceMedia work

• http://www.acemedia.org/aceMedia/reference/multimedia_ontology/index.html


7

List of contributors

1. National Technical University of Athens2. Istituto per la Matematica Applicata e le Tecnologie

Informatiche, Consiglio Nazionale delle Ricerche, AIM@SHAPE

3. University Rennes4. Centre for Digital Music, Queen Mary, University of London5. Lehrgebiet Multimedia und Internetanwendungen,

Informatikzentrum6. University of Southampton, UK7. Institute of Informatics & Telecommunications, N.C.S.R.

"Demokritos"8. Institute of Information Systems & Information

Management, JOANNEUM RESEARCH Forschungsgesellschaft mbH


8

List of contributors (2)

9. ISWeb, University of Koblenz

10. Freie Universitaet,Netzbasierte Informationssysteme, Institut fuer Informatik

11. 'Signals and Images' Laboratory, Institute of Information Science and Technologies "A. Faedo" (ISTI–CNR), Italian National Research Council (CNR)

12. Image Bioinformatics Research Group, Department of Zoology, University of Oxford

13. Technical University of Crete,

14. CWI Amsterdam, Maryland Information and Network Dynamics Lab, National Technical University of Athens

15. University of Twente


9

Requirements for MM ontologies

• Scope and Usage• Media Description-Description of the

information object• Representation of Content Structure• Representation of Content Description• Knowledge Representation Formalism• Reasoning Support • Following an established methodology


10

Requirements (2)

• Scope and Usage Annotation Analysis Retrieval Reasoning Personalized

filtering Meta-Modeling

• Media description Image Video 3D Graphics Audio AudioVisual Multimedia

presentation


11

Requirements (3)

• Representation of content structure Multimedia

structure representation

Spatiotemporal relation representation

Localization

• Representation of content description Low-level description Multiple audiovisual

descriptors Alignment to existing

metadata standards Basic and complex

data types


12

Requirements (4)

• Knowledge representation formalism High complexity of

multimedia modeling Compliance to W3C

standards Inferencing support OWL

• Reasoning support DL-based

reasoning Rules Uncertainty Fuzziness

• Established methodology OntoKnowledge OntoClean


13

Harmonization approaches (1)

• Modularization Clear, independent ontology modules Examples: Visual Descriptors Ontology (VDO),

Multimedia Structure Ontology (MSO), Spatio-Temporal Ontology (STO)

• Ontology cross-domain linking Alignment techniques for multimedia ontologies linking

with domain-specific ontologies Merging techniques for multimedia ontology

coordination

• Compliance to standards Semantic Web, W3C, MPEG-7, etc


14

Harmonization approaches (2)

• Harmonization of core ontologies Core ontologies cannot be simply mapped but

need to be “harmonized” Need to apply ontological modeling principles

during harmonization

• Development of a common upper ontology Starting point for new ontology construction Bridge between existing ontologies Act as the ground basis for the establishment of

an integrated ontology framework Back up the common upper ontology


BOEMIE

Multimedia annotation tool and

A proposed knowledge framework


16

M-OntoMat Annotizer

Mid-Level

Concepts

Selection of regions for mergingSemantic

labeling

Automatic segmentation

Region merging

functionality

Descriptor extractionDescriptor extraction


17

XML output


18

Knowledge framework

Core Ontology-Holistic Representation

Domain Ontology

Spatiotemporal Constraints

Modality Elements

Multimedia Content Ontology

Classifiers/ Algorithms

Features/ Multimedia Descriptor Ontology


19

Athletics_Event

High_ Jump_Event

Pole_Vault_EventLong_Jump_E

vent

Jumping_Event

Horizontal_Bar

Pole

Sandpit

Foam_Mats

Foam Mats

Mats

Still region

Multimedia object

VideoImage

Video segment

Element

Textual Element

Visual Element

Horizontal_Bar_ve

Sandpit_ve

Foam_Mats_ve

is_Right

is_Below

is_On

HasVis

ualEle

ment

Still_region_Inst1Foam_Mats_ve_Inst1 Still_region_Inst2

Horizontal_Bar_ve_Inst1depicts

Horizontal_Bar_Inst_1 Foam_Mats_Inst_1HasVisualElem

ent

depicts

Image_Inst1

Athletics_Event_Inst_1

Pole_Vaulter_Failed_Landing

High_Jumper_Failed_Landing

High_Jumper_Successful _Landing

Pole_Vaulter_Successful_Landing

Pole_Vaulter_Jump_Off

High_Jumper_ApproachHigh_Jumper_Jump_Off

Pole_Vaulter_Swing_and_Row

An indicative example

informatics and telematics institute centre for research and technology hellas iti-certh amsterdam,...

Documents

multimedia reasoning

retrieval multimedia

multimedia features

telematics institute

multimedia semantics

research fellow centre

telematics institute

knowledge sharing