Enabling access to Linked Media with
Thomas Kurz, Kai Schlegel, Prof. Dr. Harald Kosch
3rd International Workshop on Linked Media, WWW 2015, Florence
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
The MICO Project and me
• www.mico-project.eu• 36 Months Project• started 10/13• 3.4 Million Euro Project • 7 Partners
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
The MICO project idea
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
The MICO project idea
Semantic Web Multimedia
Linked Media
Media Fragment Identifiers
Media Annotation Ontologies
5
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
Multimedia Retrieval on the Web of Data is finding (fragments of) resources of an structured and unstructured nature (text, image, video,
concepts etc.) that satisfy an information need.
Extension of the standard definition of information retrieval in:
Christopher D. Manning and Prabhakar Raghavan. An Introduction to Information Retrieval. Cambridge University Press, 2009.
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
Multimedia Query Languages
1. SQL-like approaches (WebSSQL or SQL/MM)
2. Object oriented approaches (MOQL or POQLMM)
3. Focusing XML metadata (MMDOC-QL or XQuery)
4. Visual Query Languages, like (MQuery Visual-MOQL)
5. Query-By-Example approaches, like WS-QBE
6. Metadata agnostic like MPQF
A Survey with ~40 MM Query Languages is coming soon …
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
Multimedia Query Languages:Requirements and Features
• Spatial Operation
• Temporal Operation
• Freetext Search
• Functions: Aggreg., Sort., Group., Set-Fnct.
• Similarity Search (Top-k)
• Event / Object Concept
• Presentation Part
General Requirements:
Application Independency, Formal Semantics, Extendability, Closed, etc.
Specific Features:
• Fuzzy terms
• Relevance Feedback
• Camera Motion
• Object Motion (Trajectory)
• Query by Example (Image, Video, Sketch, Humming)
• Condition Weighting
• Low-Level-Features (Audio, Video)
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
Semantic Web Query Languages
• There has been many in the past, but SPARQL
(SPARQL Query Language for RDF) is the winner
of the "natural selection".
• With SPARQL 1.1 many missing features have
become part of the standard.
• SPARQL is pretty good to extend (even without
changing the basic syntax)
BUT: It's missing some important features
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
What we want to reach
"Find scenes where Barack Obama is left beside the Greenpeace Managing Director during an UN climate conference ordered by length."
Features needed:
• Object Identification
• Temporal Operation
• Spatial Operation
• Freetext Search
• Sorting by Temporal Selector
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
How could it look like?"Find scenes where Barack Obama is left beside the Greenpeace MD during the UN climate change summit ordered by length."SELECT ?scene WHERE {
?a3 oa:hasBody ?event; oa:hasTarget ?s3. # there are resources about an event
?event schema:Event; schema:summary ?description.# which has a description
FILTER mm:fulltext-search(str (?description), "UN climate conference","en")
# about 'UN climate conference'
?a2 oa:hasBody :Barack_Obama; oa:hasTarget ?s2. # and there are resources about Obama
?a1 oa:hasBody ?p1; oa:hasTarget ?s1. # and there are resources
?p1 :ceo_of dbpedia:Greenpeace. # about the MD of Greenpeace.
FILTER mm:leftBeside(?s2,?s1) # Obama has to be at the left of the MD and
FILTER mm:intersects(?s3,?s2) # has to be appear at the same time like the event.
BIND (mm:boundingBox(?s1,?s2) AS ?scene) # Wrap the results to scenes
ORDER BY DESC(mm:duration(?scene)) # and show the longest first.
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
Goals• Bring Multimedia Functionalities to SPARQL
(especially spatio-temporal support)
• Give transparent access to common Metadata Models (Web Annotation Selectors, Media Fragment URIs)
• Provide efficient evaluation
• Make it more easy for users
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
Spatio-Temporal Object Model
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
Spatio-Temporal Object Model: Example: Rectangle
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
Supported Standards
• Media Fragment URIs
• Spatial Fragment (currently Rectangle)
• Temporal Fragment (currently Interval)
• Web Annotation Model
• Media Fragment Selectors (http://www.w3.org/TR/media-frags/)
• SVG Basic Shapes (http://www.w3.org/TR/SVG/shapes.html)
http://example.org/video.mp4#t=1,5&xywh=10,20,30,30
© Salzburg Research Forschungsgesellschaft 2012. No reproduction without written permission. Certified in accordance with ISO 9001:2008
SPARQL-MM FunctionsSpatial Topological Relations
Spatial Predicatesbased in the
Dimensionally Extended nine-Intersection Model
+(DE-9IM)
© Salzburg Research Forschungsgesellschaft 2012. No reproduction without written permission. Certified in accordance with ISO 9001:2008
SPARQL-MM FunctionsSpatial Directional Relations
© Salzburg Research Forschungsgesellschaft 2012. No reproduction without written permission. Certified in accordance with ISO 9001:2008
SPARQL-MM Functions: Temporal Relations
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
Multimedia Tests and Accessors (not complete yet)xsd:boolean mf:isMediaFragmentURI(Resource r)
xsd:boolean mf:isMediaFragmentSelector(Resource r)
xsd:boolean mf:isSVGSelector(Resource r)
mo:Rectangle mf:getBoundingBox(mo:SpatialEntity e)
xsd:double mf:getArea(mo:SpatialEntity e)
mo:Point mf:getXY(mo:Rectangle r)
xsd:double mf:getWidth(mo:Rectangle r)
xsd:double mf:getWidth(mo:Rectangle r)
xsd:double mf:getDuration(mo:TemporalEntity e)
xsd:double mf:getStart(mo:TemporalEntity e)
xsd:double mf:getEnd(mo:TemporalEntity e)
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
Fulltext SearchBased in Apache Marmotta Fulltext Search*
• mf:fulltext-search(text, query, [language]): searches “text” for the words occurring in “query”, optionally applying the language-specific processing for the given language; query is a simple text literal (list of words) without any boolean connectors; words are AND connected, i.e. all words have to be found in the text for a successful match.
• mf:fulltext-query(text, query, [language]): searches “text” using the boolean query string passed in “query”, optionally applying language-specific processing for the given language; query is a boolean query string following the syntax used by PostgreSQL.
* http://marmotta.apache.org/kiwi/sparql.html
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
Status of Implementation
• SPARQL-MM 1.0 is out for a while, but:
• it is slow (based on Sesame Function Extensions)
• it is reduced
• it is very prototype (therefore still Snapshot)
https://oss.sonatype.org/content/repositories/snapshots/com/github/tkurz/sparql-mm/
• SPARQL-MM 2.0 will be available soon (planned for 8/2015)
• it is much faster (based on SQL-MM using PostGIS)
• it will be available as Open Source Marmotta Module*
• is has a big feature set
* maybe we should create a LIME Module that include implementations for various Linked Media related things like Web Annotation endpoints etc. ?
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
Short Demo <repositories> <repository> <id>oss-sonatype</id> <name>oss-sonatype</name> <url>
https://oss.sonatype.org/content/repositories/snapshots/</url>
</repository> </repositories>
<dependency> <groupId>com.github.tkurz</groupId> <artifactId>sparql-mm</artifactId> <version>1.0-SNAPSHOT</version> </dependency>
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
Future Plans
• SPARQL like for shapesmm:like(mo:Circle(20,20,100), ?f)
• Extend Model to Layersmm:behind(?f1, ?f2)
• SPARQL like for Resourcesmm:like(?u1, ?u2, 0.4)
© Salzburg Research Forschungsgesellschaft 2015. No reproduction without written permission. Certified in accordance with ISO 9001:2008
Thanks for your Attention!
Thomas KurzKnowledge and Media TechnologiesSalzburg Research Forschungsgesellschaft m.b.H.Jakob-Haringer-Straße 5/III | Salzburg, AustriaTel. +43 662 2288-253| Fax +43 662 [email protected]