multimedia information retrieval on a very large scale · i sensed a scream passing through nature;...
TRANSCRIPT
Fabrizio [email protected]
2016-03-04
Multimedia Information Retrievalon a Very Large Scale
Introduction
2
Overview
IntroductionFabrizio Falchi
1. Visual FeaturesFabrizio Falchi
2. Indexing for Similarity SearchGiuseppe Amato
3. Large Scale CBIR using Standard Text Retrieval EnginesClaudio Gennaro
3
Multimedia Information Retrieval
• The process of
o searching for and finding multimedia documents
• The corresponding research field is concerned with
o building the best possible multimedia search engines.
• The intriguing bit here is that
o the query itself can be a multimedia excerpt.
[“Multimedia Information Retrieval”, Stefan Rüger 2009]
4
Multimedia (adj)
• Of art, education etc.:
using more than one medium of expression or communication
• Of computer applications:
incorporating audio and video, especially interactively
5
Multimedia documents
• Consist of multimedia data
(text, images, audio, video, etc.)
• Are semistructured, i.e., contain
o structured data (e.g., metadata)
o unstructured data (e.g., text, images, audio, video, etc.)
[“Multimedia Information Retrieval”, Stefan Rüger 2009]
6
Multimedia retrieval
• There are basically two options
o Metadata based
• Multimedia documents are described by metadata
• Search is performed on metadata
• Metadata can be generated manually or automatically
o Similarity based (often called content based)
• Mathematical descriptions of media content is generated
• Retrieval is performed by searching for similar mathematical
descriptions
• Automatic medatadata generation
o Obtained leveraging on classification techniques
7
Change of the Search Paradigm
7
• Traditional YES-NO keyword search will not suffice - sortable
domains of data (numbers, strings) are assumed
• New types of data need gradual comparison and/or ranking
based on:
o similarity,
o dissimilarity,
o proximity,
o distance, closeness, etc.
8
Similarity Search
Focus on:
• efficient ways
• to locate user-relevant information in collections of objects,
• the similarity of which is quantified using a
pairwise distance measure
9
Image Similarity Search Problem
9
image database
10
Feature-based Approach
10
image layer
R
B
G
feature layer
11
Library
12
Library Catalogue
13
Catalog card
14
How did/do we search for content?
• Content was in book stored in libraries
• We use(d) card catalog containing metadata
• Metadata can be:
o Structural: data about the containers of data
o Descriptive: about the data content
• Nowadays we usually search in the (text) content
(e.g., web search engines)
15
Google Books
16
Google Books
17
How would you search for the name of this Library?
18
Results
19
Same images, different sizes
20
Guessed text
21
Guessed text
22
What’s that?
23
The Scream, Edvard Munch
• The file at http://upload.wikimedia.org/.../475px-The_Scream.jpg
24
The Scream, Edvard Munch
• The file at http://upload.wikimedia.org/.../475px-The_Scream.jpg
• One of the files of the same picture as
25
The Scream, Edvard Munch
• The file at http://upload.wikimedia.org/.../475px-The_Scream.jpg
• One of the files of the same picture
• Almost the same as
26
The Scream, Edvard Munch
• The file at http://upload.wikimedia.org/.../475px-The_Scream.jpg
• One of the files of the same picture
• Almost the same
• A picture of the object at National Gallery, Oslo as
27
The Scream, Edvard Munch
• The file at http://upload.wikimedia.org/.../475px-The_Scream.jpg
• One of the files of the same picture
• Almost the same
• A picture of the object at National Gallery, Oslo
• One of “The Scream” by Edvard Munch as
28
The Scream, Edvard Munch
• The file at http://upload.wikimedia.org/.../475px-The_Scream.jpg
• One of the files of the same picture
• Almost the same
• A picture of the object at National Gallery, Oslo
• One of “The Scream”s by Edvard Munch
• A painting by Edvard Munch as
29
The Scream, Edvard Munch
• The file at http://upload.wikimedia.org/.../475px-The_Scream.jpg
• One of the files of the same picture
• Almost the same
• A picture of the object at National Gallery, Oslo
• One of “The Scream”s by Edvard Munch
• A painting by Edvard Munch
• One of “The Scream”s by various artists as
30
The Scream, Edvard Munch
• The file at http://upload.wikimedia.org/.../475px-The_Scream.jpg
• One of the files of the same picture
• Almost the same
• A picture of the object at National Gallery, Oslo
• One of “The Scream”s by Edvard Munch
• A painting by Edvard Munch
• One of “The Scream”s by various artists
• An expressionist painting as
31
The Scream, Edvard Munch
• The file at http://upload.wikimedia.org/.../475px-The_Scream.jpg
• One of the files of the same picture
• Almost the same
• A picture of the object at National Gallery, Oslo
• One of “The Scream”s by Edvard Munch
• A painting by Edvard Munch
• One of “The Scream”s by various artists
• An expressionist painting
• A painting as
32
The Scream, Edvard Munch
• The file at http://upload.wikimedia.org/.../475px-The_Scream.jpg
• One of the files of the same picture
• Almost the same
• A picture of the object at National Gallery, Oslo
• One of “The Scream”s by Edvard Munch
• A painting by Edvard Munch
• One of “The Scream”s by various artists
• An expressionist painting
• A painting
• An hand made object as
33
The Scream, Edvard Munch
• The file at http://upload.wikimedia.org/.../475px-The_Scream.jpg
• One of the files of the same picture
• Almost the same
• A picture of the object at National Gallery, Oslo
• One of “The Scream”s by Edvard Munch
• A painting by Edvard Munch
• One of “The Scream”s by various artists
• An expressionist painting
• A painting
• An hand made object
• An artificial objectbeing the product of intentional human manufacture
34
Recognition and Semantic
• Represents Valhallveien, above Oslo
35
Painting meaning?
• “I stopped and looked out over the fjord—the sun was setting, and the clouds
turning blood red. I sensed a scream passing through nature; it seemed to me
that I heard the scream. I painted this picture, painted the clouds as actual
blood. The color shrieked. This became The Scream.” (Edvard Munch)
• Reddish sky in the background is the artist's memory of the effects of the
powerful volcaniceruption of Krakatoa
• The imagery of The Scream has been compared to that which an individual
suffering from depersonalization disorder experiences, a feeling of distortion of
the environment and one's self, and also facial pain.
• "Whistler's Mother, Wood's American Gothic, Leonardo da Vinci's Mona
Lisa and Edvard Munch's The Scream have all achieved something that most
paintings—regardless of their art historical importance, beauty, or monetary
value—have not: they communicate a specific meaning almost immediately
to almost every viewer. (Martha Tedeschi)
Wikipedia
36
The Scream, Edvard Munch
• The file at http://upload.wikimedia.org/.../475px-The_Scream.jpg
• One of the files of the same picture
• Almost the same
• A picture of the object at National Gallery, Oslo
• One of “The Scream”s by Edvard Munch
• A painting by Edvard Munch
• One of “The Scream”s by various artists
• An expressionist painting
• A painting
• An hand made object
• An artificial object
being the product of intentional human manufacture
Low-level features
High-level semantic
37
The Scream, Edvard Munch
• The file at http://upload.wikimedia.org/.../475px-The_Scream.jpg
• One of the files of the same picture
• Almost the same
• A picture of the object at National Gallery, Oslo
• One of “The Scream”s by Edvard Munch
• A painting by Edvard Munch
• One of “The Scream”s by various artists
• An expressionist painting
• A painting
• An hand made object
• An artificial object
being the product of intentional human manufacture
Classification
Matching
Recognition
38
Related Research Fields
• Computer Vision
o To understand what is in a visual content the device have to “see”
• Multimedia Information Retrieval
o To retrieve visual content from a huge datasets
(it is not feasible to “see” everything online, even for the computers)
• Data Mining
o How to extract knowledge from the visual content we have
39
Matching
• Dictionary:
o to equal; be equal to
• In some terms, the match have to be exact
• Can often be done using signature
• Examples:
o Copy detection
40
Classification (of the visual content)
Contains:
• Red/orange sky
• 3 Humans
• Road
• Oslo
• 2 coves
• Cathedral
41
Tag
Tags:
• Edvard Munch
• The Scream
• Norway
• Oslo
• Scream
• Art robbery
• Munchmuseet
• Oil
• Tempera
• Pastel
• Cardboard
• Skrik
• …
An index term assigned to a piece of information; a type of meta-
information that captures knowledge about an information resource
42
Flickr Automatic Tagging
https://www.flickr.com/photos/fabriziofalchi/2810872125
43
44
MIR for Augmented Reality
45
MIR for Tourism
http://www.visitotuscany.it/
To provide tourists with immediate access to information related to
monuments and artworks
46
Smart Ticketing
To provide citizens with effective ways to get information related to events
and making the booking process quick and easy
47
Smart shopping/marketing
To provide effective service in supermarkets by rising the efficiency of total
supply chain through quick billing and promotion of products.