melt - semantics 2020...data and web science group, university of mannheim...
TRANSCRIPT
![Page 1: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/1.jpg)
MELTMatching EvaLuation Toolkit
SEMANTiCS 2019 – Karlsruhe – 2019/09/11
Sven Hertling; Jan Portisch; Heiko Paulheim
1
![Page 2: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/2.jpg)
Joint Work
2
Heiko PaulheimData and Web Science Group, University of Mannheim
Sven HertlingData and Web Science Group, University of Mannheim
Jan PortischData and Web Science Group, University of Mannheim / SAP SE
![Page 3: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/3.jpg)
Agenda
• Motivation
• What is MELT?
• Usage Example
• Q&A
3
![Page 4: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/4.jpg)
MOTIVATION
4
![Page 5: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/5.jpg)
Ontology Alignment Evaluation Initiative (OAEI)
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 5
Ontology Alignment Evaluation Initiative• running campaigns since 2005• structured in tracks (similar to task
sets)• researchers submit their
implementation• centrally evaluated by track
organizers• results published
![Page 6: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/6.jpg)
Tooling
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 6
Semantic Evaluation at Large Scale• (among others) packaging and
evaluation• OAEI support since 2010
Holistic Benchmarking of Big Linked Data • (among others) packaging and
evaluation• OAEI support since 2017• OAEI 2018: 6/19 matchers support
HOBBIT
![Page 7: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/7.jpg)
Pain Points
• Limited evaluation capabilities in SEALS, HOBBIT, and the Alignment API
• No easy-to-use parameter tuning
• Packaging process might be complicated for new entrants to the community
• Tooling Java-focused (no Python)
• Implementation of the Alignment API not maven-based
• Tool breaks
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 7
![Page 8: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/8.jpg)
What is MELT?
8
![Page 9: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/9.jpg)
What is MELT?
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE
9
• Facilitate matcher packaging• Facilitate matcher
submission
• Easy matcher development• Non-Java matcher
development• Maven support
• Advanced evaluation capabilities • Evaluation before packaging• Allow for interactive visualization
• Allow for parameteroptimization
• Streamlined development process• Integration with existing tooling• OAEI support• Extensibility
![Page 10: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/10.jpg)
What is MELT?
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 10
Matcher Development
Matcher Fine-Tuning
Matcher Evaluation
Matcher Submission
![Page 11: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/11.jpg)
What is MELT?
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 11
Matcher Development
Matcher Fine-Tuning
Matcher Evaluation
Matcher Submission
![Page 12: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/12.jpg)
Yet Another Alignment API (YAAA)
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 12
![Page 13: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/13.jpg)
Full SEALS/HOBBIT Support
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 13
![Page 14: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/14.jpg)
External Matching
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 14
• Simple wrapping
• Packageable for HOBBIT and SEALS
• Matcher can still be evaluated in MELT
• Documentation and demo project available on GitHub
![Page 15: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/15.jpg)
What is MELT?
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 15
Matcher Development
Matcher Fine-Tuning
Matcher Evaluation
Matcher Submission
![Page 16: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/16.jpg)
Parameter Tuning
• Run matcher configurations in parallel (i.e., multi-threaded)
• Hand over ExecutionResultSet to Evaluator and pick best value according to what you want to optimize
• Out-of-the-box classes that assist you
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 16
![Page 17: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/17.jpg)
What is MELT?
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 17
Matcher Development
Matcher Fine-Tuning
Matcher Evaluation
Matcher Submission
![Page 18: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/18.jpg)
Evaluation
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 18
Track track = TrackRepository.Multifarm.getSpecificMultifarmTrack("ar", "cn");
ExecutionResultSet ers = new ExecutionResultSet();
ers.addAll(Executor.run(track.getTestCases(), new Matcher(), "Matcher"));
EvaluatorCSV evaluatorCSV = new EvaluatorCSV(ers);
evaluatorCSV.write();
![Page 19: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/19.jpg)
Evaluation
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 19
Track track = TrackRepository.Multifarm.getSpecificMultifarmTrack("ar", "cn");
ExecutionResultSet ers = new ExecutionResultSet();
ers.addAll(Executor.run(track.getTestCases(), new Matcher(), "Matcher"));
EvaluatorCSV evaluatorCSV = new EvaluatorCSV(ers);
evaluatorCSV.write();
Full OAEI support: All tracks available (one-time automated download)
![Page 20: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/20.jpg)
Evaluation
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 20
Track track = TrackRepository.Multifarm.getSpecificMultifarmTrack("ar", "cn");
ExecutionResultSet ers = new ExecutionResultSet();
ers.addAll(Executor.run(track.getTestCases(), new Matcher(), "Matcher"));
EvaluatorCSV evaluatorCSV = new EvaluatorCSV(ers);
evaluatorCSV.write();
Full OAEI support: All tracks available (one-time automated download)
Multiple evaluators available, extensible.
![Page 21: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/21.jpg)
Exemplary Evaluation
“Show me the false positive class-class mappings for Multifarm on track en-de for matcher WiktionaryMatcher.”
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 21
![Page 22: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/22.jpg)
Exemplary Evaluation
“Show me the false positive class-class mappings for Multifarm on track en-de for matcher WiktionaryMatcher.”
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 22
![Page 23: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/23.jpg)
Exemplary Evaluation
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 23
“Show me the false positive class-class mappings for Multifarm on track en-de for matcher WiktionaryMatcher.” → Just filter the correspondences!
![Page 24: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/24.jpg)
More Evaluation
Filter for Scores or Correspondences
• Micro Average Precision and Macro Average Precision over track Conference
• All residual true positives for track Anatomy
• Macro Average Class-F1 for all tracks
• …
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 24
![Page 25: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/25.jpg)
What is MELT?
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 25
Matcher Development
Matcher Fine-Tuning
Matcher Evaluation
Matcher Submission
![Page 26: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/26.jpg)
Matcher Submission
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 26
pom
xml
(fill out a template –no maven wizardry required)
> mvn install
> mvn deploy
• Creates SEALS zip file → can be submitted right away
• Creates HOBBIT docker container
• Uploads and deploys container directly in HOBBITautomatically
![Page 27: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/27.jpg)
Usage Example
27
![Page 28: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/28.jpg)
Usage Example
Motivation
“Which matchers are interesting candidates for combination?”
→ Matchers with a high F1 score and high “diversity”.
28MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE
![Page 29: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/29.jpg)
Quantitative Analysis I
Analysis OAEI 2018 results for Conference and Anatomy: Jaccard overlap of Alignments rendered as heatmap in LaTex.
Executor.loadFromAnatomyResultsFolder(“myPath”);
// few lines of other code (available on GitHub as
// example)
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 29
![Page 30: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/30.jpg)
Results for Anatomy
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 30
![Page 31: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/31.jpg)
Results for Conference
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 31
![Page 32: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/32.jpg)
Quantitative Analysis II
Mean Absolute Deviation (MAD) of Similarities plotted against F1.
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 32
![Page 33: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/33.jpg)
Results for Anatomy
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 33
![Page 34: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/34.jpg)
Results for Anatomy
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 34
![Page 35: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/35.jpg)
Results for Conference
MOTIVATION | WHAT IS MELT? | USAGE EXAMPLE 35
![Page 36: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/36.jpg)
There is MUCH more to MELT
Ontology Caching Services
36
Matcher Pipelining
Baseline Matchers
One-Time Auto-Downloadof OAEI Tracks
Automatic Reading of OAEI Result Alignments
OAEI-Track Organizer Tools
TRY IT!
Execution of SEALS packages from within MELT
Alignment Extensions
Multi-Threaded Matcher Execution
ExecutionResultIndexing
Alignment Refiners
![Page 37: MELT - SEMANTiCS 2020...Data and Web Science Group, University of Mannheim heiko@informatik.uni-mannheim.de Sven Hertling ... Holistic Benchmarking of Big Linked Data • (among others)](https://reader033.vdocuments.site/reader033/viewer/2022060208/5f0402a97e708231d40bdf7f/html5/thumbnails/37.jpg)
Thank you!
Sven HertlingData and Web Science Group, University of [email protected]
Jan PortischData and Web Science Group, University of [email protected]
Heiko PaulheimData and Web Science Group, University of [email protected]
37