a framework for benchmarking entity-annotation systems

28
A Framework for Benchmarking Entity- Annotation Systems Source WWW’13 Authors Macro Cornolti, Paolo Ferrag and Massimiliano Advisor Dr. Jia-Ling, Koh Speaker Wei Chang

Upload: masako

Post on 23-Feb-2016

47 views

Category:

Documents


1 download

DESCRIPTION

A Framework for Benchmarking Entity-Annotation Systems. Source : WWW’ 13 Authors : Macro Cornolti , Paolo Ferragina and Massimiliano Advisor : Dr . Jia -Ling, Koh Speaker : Wei Chang. Outline. Introduction Entity Annotation Systems Measurement Experiment Conclusion. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: A Framework for Benchmarking Entity-Annotation  Systems

A Framework for Benchmarking Entity-Annotation Systems

Source : WWW’13Authors : Macro Cornolti, Paolo Ferragina

and MassimilianoAdvisor : Dr. Jia-Ling, KohSpeaker : Wei Chang

Page 2: A Framework for Benchmarking Entity-Annotation  Systems

Outline Introduction Entity Annotation Systems Measurement Experiment Conclusion

Page 3: A Framework for Benchmarking Entity-Annotation  Systems

Entity Annotation System

Knowledge Base

Page 4: A Framework for Benchmarking Entity-Annotation  Systems

Example

Page 5: A Framework for Benchmarking Entity-Annotation  Systems

Entity Annotation Systems

Many different entity annotation systems

But how to compare these systems?

Page 6: A Framework for Benchmarking Entity-Annotation  Systems

Goal Defining and implementing a framework for

comparing in a complete, fair and meaningful way the entity annotation systems.

Page 7: A Framework for Benchmarking Entity-Annotation  Systems

Outline Introduction Entity Annotation Systems Measurement Experiment Conclusion

Page 8: A Framework for Benchmarking Entity-Annotation  Systems

Entity Annotation Problems

1. Disambiguate to Wikipedia (D2W)2. Annotate to Wikipedia (A2W)3. Scored-annotate to Wikipedia (Sa2W)4. Concepts to Wikipedia (C2W)5. Scored concepts to Wikipedia (Sc2W)6. Ranked-concepts to Wikipedia (Rc2W)

Page 9: A Framework for Benchmarking Entity-Annotation  Systems

Entity Annotation Problems

Page 10: A Framework for Benchmarking Entity-Annotation  Systems

Entity Annotation Problems

Page 11: A Framework for Benchmarking Entity-Annotation  Systems

Entity Annotation Problems

Page 12: A Framework for Benchmarking Entity-Annotation  Systems

Entity Annotation Problems

Page 13: A Framework for Benchmarking Entity-Annotation  Systems

Outline Introduction Entity Annotation Systems Measurement Experiment Conclusion

Page 14: A Framework for Benchmarking Entity-Annotation  Systems

Correct MatchPresident Barack Obama issues Iran ultimatum

Barack Obama

Barack ObamaPresident Barack Obama issues Iran ultimatum

Page 15: A Framework for Benchmarking Entity-Annotation  Systems

Generalize Standard Evaluation

s: solution found by tested systemg: ground truthM: binary relation M which specifies the notion of correct match"

Page 16: A Framework for Benchmarking Entity-Annotation  Systems

Precision, Recall & F1

Page 17: A Framework for Benchmarking Entity-Annotation  Systems

Macro and Micro

The macro- measures are the average of the correspondingmeasure over each document in the dataset D, while the micro- measures take into account all annotations together thus giving more importance to documents with more annotations.

Page 18: A Framework for Benchmarking Entity-Annotation  Systems

Correct MatchIf the entity found by the system and the ground truth entity redirect to the same entity, then set the correctness true.

The mention and the entity correct must be right(strong annotation match).

The mention overlap and the entity is right(weak annotation match).

Page 19: A Framework for Benchmarking Entity-Annotation  Systems

Similarity between Systems

Page 20: A Framework for Benchmarking Entity-Annotation  Systems

Outline Introduction Entity Annotation Systems Measurement Experiment Conclusion

Page 21: A Framework for Benchmarking Entity-Annotation  Systems

Dataset

Page 22: A Framework for Benchmarking Entity-Annotation  Systems

News

Page 23: A Framework for Benchmarking Entity-Annotation  Systems

Similarity between Systems

Page 24: A Framework for Benchmarking Entity-Annotation  Systems

Tweet

Page 25: A Framework for Benchmarking Entity-Annotation  Systems

Web

Page 26: A Framework for Benchmarking Entity-Annotation  Systems

Runtime Efficiency

Page 27: A Framework for Benchmarking Entity-Annotation  Systems

Outline Introduction Entity Annotation Systems Measurement Experiment Conclusion

Page 28: A Framework for Benchmarking Entity-Annotation  Systems

Conclusion We designed, implemented and tested a

benchmarking framework to fairly and fully compare entity-annotation systems.

It is written in Java and it has been released to the public as open source code in https://github.com/marcocor/bat-framework .