multimedia information retrieval · kmi.open.ac.uk since 1995: 117 projects & 67 technologies...
TRANSCRIPT
![Page 1: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/1.jpg)
Multimedia Information Retrieval
Prof Stefan Rüger
Multimedia and Information SystemsKnowledge Media Institute
The Open Universityhttp://kmi.open.ac.uk/mmis
![Page 2: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/2.jpg)
kmi.open.ac.uk
![Page 3: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/3.jpg)
kmi.open.ac.uk
![Page 4: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/4.jpg)
kmi.open.ac.uk
Since 1995: 117 projects & 67 technologies
Current year
17 live projects , typically per year£2.5m (¥300m) ext, £1m (¥120m) internal• 10 EU• 3 UK • 1 US• 3 internal (iTunes U, SocialLearn)
![Page 5: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/5.jpg)
Multimedia Information Retrieval
1. What are multimedia queries?
2. Fingerprinting
3. Metadata & piggy-back retrieval
4. Automated image annotation
5 Visual content-based retrieval I
6 Visual content-based retrieval II
7. Evaluation
8. Browsing, search and geography
![Page 6: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/6.jpg)
Multimedia Information Retrieval
1. What are multimedia queries? - What is multimedia? - Query by image - Current best practice for image search - Snaptell/Google goggles - Shazam - Discussion: Challenges and difficulties
2. Fingerprinting
3. Metadata & piggy-back retrieval
4. Automated image annotation
5 Visual content-based retrieval I
6 Visual content-based retrieval II
7. Evaluation8. Browsing, search and geography
![Page 7: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/7.jpg)
What is Multimedia?
Within this lecture:One or more mediaPossibly interlinkedDigitalFor communication (not only entertainment)
![Page 8: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/8.jpg)
Sensō-ji ( � � � � � � Kinryū-zan Sensō-ji?) is an ancient Buddhist templelocated in Asakusa, Taitō, Tokyo, Japan. It is Tokyo's oldest temple, and one of its most significant. Formerly associated with the Tendai sect, it became independent after World War II. Adjacent to the temple is a Shinto shrine,the Asakusa Shrine [http://en.wikipedia.org/wiki/Sensō-ji]
Multimedia queries
![Page 9: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/9.jpg)
Web-based image searching
“Tokyo temple”
Google ImagesBing ImagesFlickrYahoo ImagesЯндекс
![Page 10: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/10.jpg)
Web-based image searching
Best current practice is a text search:Find text in filename, anchor text, caption, ...
Text search works by creating a large index:
![Page 11: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/11.jpg)
GoogleTokyo temple
![Page 12: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/12.jpg)
BingTokyo temple
![Page 13: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/13.jpg)
FlickrTokyo temple
![Page 14: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/14.jpg)
YahooTokyo temple
![Page 15: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/15.jpg)
YandexTokyo temple
![Page 16: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/16.jpg)
New search types
query doc
conventional text retrieval
hum a tune and get a music piece
you roar and get a wildlife documentarytype “floods” and get BBC radio news
Example
text
video
images
speech
music
sketches
multimedia
loca
tion
sound
hum
min
g
mot
ion
text
imag
e
spee
ch
![Page 17: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/17.jpg)
Exercise
Organise yourself in groupsDiscuss with neighbours - Two Examples for different query/doc modes? - How hard is this? Which techniques are involved? - One example combining different modes
![Page 18: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/18.jpg)
Exercise
query doc
Discuss
- 2 examples
- How hard is it?
- 1 combination
loca
tion
sound
hum
min
g
mot
ion
text
imag
e
spee
ch
loca
tion
sound
hum
min
g
mot
ion
text
imag
e
spee
ch
text
video
images
speech
music
sketches
multimedia
![Page 19: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/19.jpg)
Near-duplictate detection:Cool access mode!
![Page 20: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/20.jpg)
Snaptell: Book, CD and DVD covers
![Page 21: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/21.jpg)
Snaptell: Book, CD and DVD covers
![Page 22: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/22.jpg)
Snaptell: Book, CD and DVD covers
![Page 23: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/23.jpg)
Snaptell: Book, CD and DVD covers
![Page 24: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/24.jpg)
Snaptell: Book, CD and DVD covers
![Page 25: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/25.jpg)
Link from real world to databases
doi: 10.2200/S00244ED1V01Y200912ICR010
![Page 26: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/26.jpg)
The Open Univerity'sSpot & Search
Scott Forrest: E=MC squared
"Between finished surface texture and raw quarried stone. Between hard materials and soft concepts.
Between text and context."
More information
[with Suzanne Little]
![Page 27: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/27.jpg)
Spot & Search
[with Suzanne Little]
![Page 28: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/28.jpg)
Near duplicate detection
Works well in 2d: CD covers, wine labels, signs, ...Less so in near 2d: buildings, vases, …Not so well in 3d: faces, complex objects, ...
![Page 29: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/29.jpg)
Shazam
Rueger, Multimedia IR, 2010explains it all! Buy it now
![Page 30: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/30.jpg)
Near duplicate detectionExercise
Find applications for near-duplicate detection - be imaginative: the more “outragous” the better - can be other media types (audio, smells, haptic, ...) - can be hard to do
![Page 31: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/31.jpg)
Near-duplicate detectionWhere are the challenges?
[Victoria and Albert museum, London, ceramics collection, 2010]
![Page 32: Multimedia Information Retrieval · kmi.open.ac.uk Since 1995: 117 projects & 67 technologies Current year 17 live projects , typically per year £2.5m (¥300m) ext, £1m (¥120m)](https://reader035.vdocuments.site/reader035/viewer/2022081406/5f154e42bf5a000bda6f2a1b/html5/thumbnails/32.jpg)
Leaf detectionWhat are the challenges?
[with Natural History Museum, London, and Goldsmiths]