benchmarking the effectiveness of associating chains of links for exploratory semantic search
TRANSCRIPT
Benchmarking the Effectiveness of Associating
Chains of Links for Exploratory Semantic Search
Laurens De Vocht Selver Softic, Ruben Verborgh, Erik Mannens, Martin Ebner, Rik Van de Walle
:Paris
?
2
:Paris :Anne_Hidalgo
:mayor
3
:Paris :Anne_Hidalgo
:mayor
:Bethlehem,_PA
?
4
:Anne_Hidalgo
?
5
:Anne_Hidalgo
?
:Bethlehem,_PA
:Anne_Hidalgo
:Bethlehem,_PA
Exploratory Semantic Search Engine
7
?
8
:Paris
9 :mayor :Anne_Hidalgo
< :birthPlace :San_Fernando,_Caldiz
9 :country :Spain
< :birthPlace :Edward_Ferrero
9 :battle :Battle_of_Roanoke_Island
< :battle :Charles_Adam_Heckman
9 :birthPlace :Easton,_Pennsylvania
9 :mouthMountain :Lehigh_River
9 :city
:Bethlehem,_Pennsylvania
A
9
:Paris
< :capital :France
< :citizenship :Cyril_Bourlon_de_Rouvre
9 :education :Aerospace_engineering
< :occupation :Dick_Johnson_(glider_pilot)
9 :almaMater :Mississippi_State_University
< :almaMater :Clara_Southmayd_Ludlow
9 :birthPlace :Easton,_Pennsylvania
< :mouthMountain :Lehigh_River
9 :city
:Bethlehem,_Pennsylvania
B
10
B A
?
How effective does an exploratory semantic
search engine reveal initially hidden associations,
as chains of links between interlinked resources?
Introduction
Exploratory Search
Benchmark Model
Motivating Example
Discussion and Conclusion
Introduction
Exploratory Search
Benchmark Model
Motivating Example
Discussion and Conclusion
[EXPLORATORY SEARCH: FROM FINDING TO UNDERSTANDING, Machionini, 2006]
Lookup Learn Investigation
Exploratory Search
`Learning searches involve multiple iterations and return sets of
objects that require cognitive processing and interpretation’
`Searches that support investigation involve multiple iterations that take place
over perhaps very long periods of time and may return results that are critically
assessed before being integrated into personal and professional knowledge bases’
Definition
15
1. Lookup
2. Relate/Expand
Lookup and learn: interpretation
16
lookup
expand relate
An exploratory semantic search engine
17
lookup
:Paris
Paris
18
expand
:Paris
:Paris
19
relate
20
Introduction
Exploratory Search
Benchmark Model
Motivating Example
Discussion and Conclusion
Iterative Exploratory Queries
Exploratory Semantic Search Engine
Datasets Baseline
Effectiveness
22
Effectiveness
The effectiveness E indicates the overall perception of the results by
the users taking into account expert-user feedback.
# user marked relevant objects
E =
# retrieved objects
Note:
E can be interpreted as precision in traditional IR.
Typical IR examine both precision and recall.
23
[TALKEXPLORER, Verbert et al., 2013]
Introduction
Exploratory Search
Benchmark Model
Motivating Example
Discussion and Conclusion
Motivating Example
ResXplorer.org
Everything Is Connected Engine
Virtuoso
User Study Extracted Queries
25
User Study Extracted Queries
1. lookup; 2. expand; 3. relate
26
LDOW
P(0)
P(1)
P(2)
P(3)
Effectiveness : Interpretation
27
Sample Results
Everything Is Connected Engine
28
Sample Results
Virtuoso
29
Introduction
Exploratory Search
Benchmark Model
Motivating Example
Discussion and Conclusion
Limitations
Only indicate comparisons to baseline within the same use case.
Not possible to use the benchmark as a leverage to compare different
approaches across use cases
Could better demonstrate in which aspects an exploratory approach
excels traditional systems.
31
Future Work
Put the results in perspective by indicating the nuances among different
expert user ratings.
Especially when there is expert disagreement or inconsistencies.
Facilitate generalization of the preliminary search context,
so results for engines can be reusable across datasets: avoiding that a
certain engine’s results differ strongly when changing the data and queries.
Make sure that the approach is generic and can be applied to other search
contexts with different data and use cases.
32
Benefits
Compare exploratory search engines to a baseline:
show use cases when the baseline can be outperformed;
for which queries the ‘engine under test’ is relatively more effective.
Sensitive to initial query keywords as inputted by the user:
when there are inconsistencies or vague terms,
even mismatches in the query context, or when expert users disagreed.
33
Contact
@laurens_d_v
http://slideshare.net/laurensdv
http://semweb.mmlab.be/