contentus—a semantic multimodal search user interface
Post on 16-Apr-2017
1.613 Views
Preview:
TRANSCRIPT
More than the Sum of its Parts
CONTENTUS— A Semantic Multimodal Search User Interface
Jörg Waitelonis, Johannes Osterhoff, Harald Sack
Visual Interfaces to the Social and Semantic Web (VISSW 2011) In conjunction with the International Conference on Intelligent User Interfaces (IUI 2011), J. Osterhoff, February 13, 2011
CONTENTUS? A Teaser
Semantic Search Engine
Cultural Heritage Domain
Media types
Articles, Books, Video, Audio, …
People, Locations, …
Part of the HPI research project
Semantic Media Explorer
VISSW 2011 | Johannes Osterhoff | February 13, 2011
2
CONTENTUS? A Teaser
VISSW 2011 | Johannes Osterhoff | February 13, 2011
3
Issue #1
How to find the needle
in the haystack?
VISSW 2011 | J. Osterhoff | February 13, 2011
4
VISSW 2011 | J. Osterhoff | February 13, 2011
5
Image:
iSto
ckphoto
VISSW 2011 | J. Osterhoff | February 13, 2011
6
needle
VISSW 2011 | J. Osterhoff | February 13, 2011
7
Google Results
Needless to say…
8
needless
VISSW 2011 | J. Osterhoff | February 13, 2011
“needle”
VISSW 2011 | J. Osterhoff | February 13, 2011
9
VISSW 2011 | J. Osterhoff | February 13, 2011
10
Jimmy Needles American basketball coach
1900—1969
Person Image:
Jay P
. Lee
“needle”
VISSW 2011 | J. Osterhoff | February 13, 2011
11
Needles, CA
VISSW 2011 | J. Osterhoff | February 13, 2011
12
Location
Image:
Eddie
Big
gs
“needle”
VISSW 2011 | J. Osterhoff | February 13, 2011
13
“Needle”
VISSW 2011 | J. Osterhoff | February 13, 2011
14
Blockbuster by John V. Soto from 2010
Film
Image:
IMD
B
Issue #1
How to find different “needles”
in the haystack?
VISSW 2011 | J. Osterhoff | February 13, 2011
15
Issue #2
How to present
“needles”
as search results?
VISSW 2011 | J. Osterhoff | February 13, 2011
16
VISSW 2011 | J. Osterhoff | February 13, 2011
17
Google Results
VISSW 2011 | J. Osterhoff | February 13, 2011
18
Flickr Results
VISSW 2011 | J. Osterhoff | February 13, 2011
19
YouTube Results
VISSW 2011 | J. Osterhoff | February 13, 2011
20
Wikipedia Results
Issue #2
How to present
different “needles”
as distinguishable
search results
on one page?
VISSW 2011 | J. Osterhoff | February 13, 2011
21
VISSW 2011 | J. Osterhoff | February 13, 2011
22
Why?
VISSW 2011 | J. Osterhoff | February 13, 2011
23
30,000+ Libraries in Germany
VISSW 2011 | J. Osterhoff | February 13, 2011
24
30,000+ Libraries in Germany
VISSW 2011 | J. Osterhoff | February 13, 2011
25
Books
Image:
Lib
rary
of
Congre
ss
VISSW 2011 | J. Osterhoff | February 13, 2011
26
Articles
VISSW 2011 | J. Osterhoff | February 13, 2011
27
Video
Image:
Lib
rary
of
Congre
ss
VISSW 2011 | J. Osterhoff | February 13, 2011
28
Audio
Issue #3
How to make the German
cultural heritage available?
VISSW 2011 | J. Osterhoff | February 13, 2011
29
VISSW 2011 | J. Osterhoff | February 13, 2011
30
Authors i.e. Bertolt Brecht
VISSW 2011 | J. Osterhoff | February 13, 2011
31
Years i.e. 1848
People i.e. from Ulm
VISSW 2011 | J. Osterhoff | February 13, 2011
33
Locations among many others…
Image:
Wik
ipedia
(Soft
eis
)
VISSW 2011 | J. Osterhoff | February 13, 2011
34
Music Style
Image:
Wik
ipedia
VISSW 2011 | J. Osterhoff | February 13, 2011
35
Process Chain
HPI Research
Books, Articles, Video, Audio
Authors, Year of Creation,
People, Locations, Music Styles,
Organizations etc.
VISSW 2011 | J. Osterhoff | February 13, 2011
36
Media Assets
Cultural Categorizations
On the Interface
Issue #3
How to make the German
cultural heritage available
in meaningful
information retrieval?
VISSW 2011 | J. Osterhoff | February 13, 2011
37
VISSW 2011 | J. Osterhoff | February 13, 2011
38
VISSW 2011 | J. Osterhoff | February 13, 2011
39
1. What is CONTENTUS?
2. Interface Design
3. Demonstration of the System
4. Summary
What is CONTENTUS? Introduction
■ a semantic search engine and a multimedia
retrieval system
VISSW 2011 | Johannes Osterhoff | February 13, 2011
40
What is CONTENTUS? Introduction
■ a semantic search engine and a multimedia
retrieval system
■ web-based graphical user interface
VISSW 2011 | Johannes Osterhoff | February 13, 2011
41
What is CONTENTUS? Introduction
■ a semantic search engine and a multimedia
retrieval system
■ web-based graphical user interface
■ content-based suggestions and
a multi-modal presentation of
search results to support semantic search
VISSW 2011 | Johannes Osterhoff | February 13, 2011
42
What is CONTENTUS? Introduction
■ a semantic search engine and a multimedia
retrieval system
■ web-based graphical user interface
■ content-based suggestions and
a multi-modal presentation of
search results to support semantic search
■ developed in the context of the German
research project THESEUS
VISSW 2011 | Johannes Osterhoff | February 13, 2011
43
What is CONTENTUS? Introduction
■ Combination of a facetted browsing and
breadcrumb-based navigation
■ A time-line enables time-based filtering
of search results
■ Personalized search (in the next demonstrator)
VISSW 2011 | Johannes Osterhoff | February 13, 2011
44
What is CONTENTUS? Introduction
“More than the sum of its parts”
VISSW 2011 | J. Osterhoff | February 13, 2011
45
What is CONTENTUS? Introduction
More efficient
and more pleasant
user experience.
VISSW 2011 | J. Osterhoff | February 13, 2011
46
What is CONTENTUS? Context
■ new technologies for managing,
organizing, and searching multimedia
content gets more efficient
■ tremendous diversity of multimedia
data affects the retrieval and search
process
■ image and video search technology is
evolving rapidly
VISSW 2011 | Johannes Osterhoff | February 13, 2011
47
What is CONTENTUS? Context
VISSW 2011 | J. Osterhoff | February 13, 2011
48
■ semantic search technology promises
more accurate and complete search
results
■ new user interfaces
are required to enable query formulation
and intelligent visualization of search
results
What is CONTENTUS? Technologies
VISSW 2011 | Johannes Osterhoff | February 13, 2011
49
■ all types of media assets
■ technologies
media optimization
content analysis
semantic metadata generation
speech transcripts
OCR
extraction of semantic entities (i.e. people, places, events)
■ highly heterogeneous collection of semantically
annotated data
VISSW 2011 | J. Osterhoff | February 13, 2011
50
1. What is CONTENTUS?
2. Interface Design
3. Demonstration of the System
4. Summary
User Interface Paradigms
No research on
a single aspect of
current approaches
in semantic multimedia
retrieval…
VISSW 2011 | Johannes Osterhoff | February 13, 2011
51
User Interface Paradigms
But:
exploiting synergies
by combining
state-of-the-art
technologies instead.
VISSW 2011 | Johannes Osterhoff | February 13, 2011
52
User Interface Paradigms
semantic
multimedia
technologies
VISSW 2011 | Johannes Osterhoff | February 13, 2011
53
User Interface Paradigms
semantic +
multimedia
technologies
VISSW 2011 | Johannes Osterhoff | February 13, 2011
54
User Interface Paradigms
semantic +
multimedia
technologies +
interface design
VISSW 2011 | Johannes Osterhoff | February 13, 2011
55
User Interface Paradigms
combination of multimedia
and semantic search
= new challenges in visualizing
search results and facilitating
navigation
VISSW 2011 | J. Osterhoff | February 13, 2011
56
User Interface Paradigms
compare different media at one glance
how to summarize content most
meaningfully in one single search
result item
time-based information (audio, video)
VISSW 2011 | J. Osterhoff | February 13, 2011
57
User Interface Paradigms
multimodal
VISSW 2011 | Johannes Osterhoff | February 13, 2011
58
User Interface Paradigms
multimodal
= articles, books,
video, audio, people,
locations on one
result-page
VISSW 2011 | Johannes Osterhoff | February 13, 2011
59
User Interface Paradigms
The route with the
user‟s experience
gets more important
than the original goal
of fact finding.
VISSW 2011 | Johannes Osterhoff | February 13, 2011
60
User Interface Paradigms
main goal
satisfying search experience
for the user
VISSW 2011 | J. Osterhoff | February 13, 2011
61
62
63 Search Area
64 Search Field breadcrumbs
65
66
67
Result Area
68
organize and filter
media types
history
collection
69
refine and filter
time-line
faceted suggestions
70 search results
71
article
audio
video
location
article
person video
search results
72
73
74
75
76
less important search results
77
User Interface Detail Page for Books
VISSW 2011 | Johannes Osterhoff | February 13, 2011
78
Detail View
User Interface Detail Page for Books
VISSW 2011 | Johannes Osterhoff | February 13, 2011
79
Detail View detected entities
full page
other pages
articles on this page
text snippet (OCR)
VISSW 2011 | J. Osterhoff | February 13, 2011
80
1. What is CONTENTUS?
2. Interface Design
3. Demonstration of the System
4. Summary
VISSW 2011 | Johannes Osterhoff | February 13, 2011
81
Demo http://mediaglobe.yovisto.com:8081/contentus/
VISSW 2011 | J. Osterhoff | February 13, 2011
82
1. What is CONTENTUS?
2. Interface Design
3. Demonstration of the System
4. Summary
VISSW 2011 | J. Osterhoff | February 13, 2011
83
Issue #1
How to find the needle
in the haystack?
Issue #2
How to present
“needles”
as search results?
Issue #3
How to make the German
cultural heritage available?
Issue #1
How to find different
“needles” in the haystack?
Issue #2
How to present
different “needles” as
distinguishable search
results?
Issue #3
How to make the German
Cultural Heritage available
in meaningful Information
Retrieval?
Summary
Future Work
■ Feedback that lets the user know why exactly the listed
items have been presented
■ Possibility to connect with friends and to share results
with them
■ Suggestions based on friends„ likes
■ State-of-the-art multi-platform web design
■ Next demonstrator to be presented at
IBC 2011 Amsterdam
VISSW 2011 | Johannes Osterhoff | February 13, 2011
84
VISSW 2011 | Johannes Osterhoff | February 13, 2011
85
VISSW 2011 | Johannes Osterhoff | February 13, 2011
86 Person
Location
Book
Video
Audio
Thank you!
Johannes P Osterhoff, M.A.
johannes.osterhoff@hpi.uni-potsdam.de Hasso-Plattner-Institute, Prof.-Dr.-Helmert-Str. 2-3, 14482 Potsdam, Germany
VISSW 2011 | Johannes Osterhoff | February 13, 2011
87
top related