eswc - phd symposium 2016

19
Machine-Crowd Annotation Workflow for Event Understanding across Collections & Domains Oana Inel Extended Semantic Web Conference PhD Symposium May 30 th 2016

Upload: oanainel

Post on 12-Apr-2017

1.600 views

Category:

Data & Analytics


1 download

TRANSCRIPT

Page 1: ESWC - PhD Symposium 2016

Machine-Crowd Annotation Workflow for Event Understanding

across Collections & Domains

Oana Inel Extended Semantic Web ConferencePhD Symposium

May 30th 2016

Page 2: ESWC - PhD Symposium 2016

Too much information ... e.g., if you are interested in the topic of “whaling”

Page 3: ESWC - PhD Symposium 2016

… and after a while it all looks the sameit is difficult to form a global picture on a topic

Page 4: ESWC - PhD Symposium 2016

… thus, content without context is difficult to processevents can help create context around content

Page 5: ESWC - PhD Symposium 2016

…, but events are not easy to deal with• Events are vague• Event semantics are difficult

• Events can be viewed and interpreted from multiple perspectives and interpretations

e.g. of participants interpretation: The mayor of the city called the celebration a success. • Events can be presented at different levels of granularities

e.g. of spatial disagreement: The celebration took place in every city in the Netherlands.• People are not consistent in the way they talk about or use events

e.g.: The celebration took place last week, fireworks shows were held everywhere.

Page 6: ESWC - PhD Symposium 2016

… a lot of ground truth is needed to learn event specifics

• Traditional ground truth collection doesn’t scale:

• there is not really ‘one type of experts’ when it comes to events

• the annotation guidelines for events are difficult to define

• the annotation of events can be a tedious process

• all of the above can result in high inter-annotator disagreement

• Crowdsourcing could be an alternative

• but is still not a robust & replicable approach

Page 7: ESWC - PhD Symposium 2016

… let’s look at some examples

According to department policy prosecutors must make a strong showing that lawyers' fees came from assets tainted by illegal profits before any attempts at seizure are made.

The unit makes intravenous pumps used by hospitals and had more than $110 million in sales last year according to

Advanced Medical.

Page 8: ESWC - PhD Symposium 2016

… here is what experts annotate on these sentences

[According] to department policy prosecutors must make a strong [showing] that lawyers' fees [came] from assets

tainted by illegal profits before any [attempts] at [seizure] are [made].

The unit makes intravenous pumps used by hospitals and [had] more than $110 million in [sales] last year according to

Advanced Medical.

Page 9: ESWC - PhD Symposium 2016

… here is what the crowd annotates on them

According to department policy prosecutors must make a [strong [showing]] that lawyers' fees [[came] from assets] [tainted] by illegal profits before any [attempts] at [seizure]

are [made].

The unit [makes] intravenous pumps [used] by hospitals and [[had] more than $110 million in [sales]] last year according to

Advanced Medical.

Page 10: ESWC - PhD Symposium 2016

… here is what the machines can detect

According to department policy prosecutors must [make] a strong showing that lawyers' fees [came] from assets

[tainted] by illegal profits before any attempts at seizure are made.

The unit [makes] intravenous pumps [used] by hospitals and [had] more than $110 million in sales last year according to

Advanced Medical.

Page 11: ESWC - PhD Symposium 2016

Research Questions

• Can crowdsourcing help in improving event detection?

• Can we provide reliable crowdsourced training data?

• Can we optimize the crowdsourcing process by using results from NLP tools?

• Can we achieve a replicable data collection process across different data types and use cases?

Page 12: ESWC - PhD Symposium 2016

Current Hypothesis:

Disagreement-based approach to crowdsource ground truth is reliable and produces quality results

Page 13: ESWC - PhD Symposium 2016

Preliminary Results - Crowd vs. Experts● 200 news snippets from TimeBank● 3019 tweets published in 2014

● potential relevant tweets for events such as ‘whaling’, ‘Davos 2014’ among others

CrowdTruth approach outperforms the-state-of-the-art crowdsourcing approaches such as single annotator and

majority vote

The crowd performs almost as good as the experts due to very linguistic-specialized guidelines for expert annotators

Page 14: ESWC - PhD Symposium 2016

Current Hypothesis:

Disagreement-based approach to crowdsource ground truth can be optimised by using results from NLP tools

Page 15: ESWC - PhD Symposium 2016

Preliminary Results - Hybrid Workflow

ENTITY EXTRACTION

EVENTS CROWDSOURCING AND LINKING TO CONCEPTS

SEGMENTATION & KEYFRAMES

LINKING EVENTS AND CONCEPTS TO KEYFRAMES

diveplus.beeldengeluid.nl

Page 16: ESWC - PhD Symposium 2016

Preliminary Results - Hybrid Workflow Outcome

diveplus.beeldengeluid.nl

Page 17: ESWC - PhD Symposium 2016

Approach: Disagreement is Signal

Principles for disagreement-based crowdsourcing

• Do not enforce agreement

• Capture a multitude of views

• Take advantage of existing tools, reuse their functionality

This results in teaching machines to reason in the disagreement space

Page 18: ESWC - PhD Symposium 2016

Overall Methodology1. Instantiate the research methodology with specific data, domain

• Video synopsis, news

2. Identify state-of-the-art IE approaches that can be used

• NER tools for identifying events and their participating entities in the video synopsis

3. Evaluate IE approaches and identify their drawbacks

• Poor performance in extracting events

4. Combine IE with crowdsourcing tasks in a complementary way

• Use crowdsourcing for identifying the events and linking them with their participating entities

5. Evaluate crowdsourcing results with CrowdTruth disagreement-first approach

• Evaluate the input unit, the workers and the annotations

6. Instantiate the same workflow with different data and/or different domain

• Tweets, Twitter

7. Perform cross-domain analysis

• Event extraction in video synopsis vs. event extraction in tweets

Page 19: ESWC - PhD Symposium 2016

Project Websites http://CrowdTruth.org

http://diveproject.beeldengeluid.nl

Tools & Codehttp://dev.CrowdTruth.org

http://github.com/CrowdTruth

http://diveplus.beeldengeluid.nl

Datahttp://data.crowdtruth.org

http://data.dive.beeldengeluid.nl