harnessing the power of machines & crowds for event extraction

Vrije Universiteit Amsterdam

Harnessing the Power of Machines & Crowds for Event Extraction

what can we learn from the crowd?

Oana Inel March 9, 2015

1


My research is about: Event extraction from different data types

My goal is: A generic machine-human workflow for event extraction

My research question is: Can diversity-driven crowdsourcing of events improve the machine extraction of events and their role fillers?

Why is it relevant to you: Events play an important role in understanding of context, influence human interpretation, and thus impact everyday interaction with applications

<MY PRESENTATION>

2


● Events play an important role in understanding of context, influence human interpretation, and thus impact everyday interaction with applications

● NLP tools are a very bad in detecting events

○ events are vague

○ events carry different perspectives for their interpretation

○ events have diverse level of granularity

○ there is a lack of training data

■ in different domains

Problem Statement

3


● Event Extraction in Video Descriptions:

○ videos from Sound and Vision

○ goal: to enrich video descriptions with event info

● Event Extraction from Tweets:○ selection of tweets from 2014 on 8 events○ goal: to identify tweets, which bring salient info on given

event

● Event Extraction from news:○ news articles from 2004-2013 WikiNews collection○ goal: to identify news articles that bring salient info on a given

event

Use Cases & Datasets

4


Video Description - Workflow Overview

5

Machine AnnotationCrowd Annotation

through CrowdTruth.org


Video Description - Example

6

Polygoon 1000ste Hollands Nieuws speech door de heer Van Staveren

Toespraak van D. van Staveren, voorzitter van de Centrale

Commissie Filmkeuring, vanachter zijn bureau te Den Haag. In zijn

toespraak kondigt hij de duizendste aflevering van

Polygoons "Hollands Nieuws" aan en spreekt hij over de

uitzonderlijke kwaliteiten, actualiteit en betrouwbaarheid van

het Polygoon-journaal. Aantekening : Locatie volgens

titellijst: Haarlem.


Video Description - Example

7

1000th Polygoon Dutch News speech by mr. Van Staveren

D. van Staveren, chairman of the Central Film Censorship Board, gives a speech from behind his

desk in The Hague. In his speech he announces the one thousandth

episode of Polygoon's "Dutch News" and praises the exceptional quality, topicality and reliability of the Polygoon newsreels. Note : Location according to title list:

Haarlem.


Different span confidence

8















confidence:0.09

confidence:0.12


Different span confidence

9















confidence:0.11 THD

confidence:null


Missed entities

10
















type:Location

Different / Misidentified Types (1)

11















THD

type:NULL


Different / Misidentified Types (2)

12















type:NULL

type:DBPedia::Place


Crowd - Determine Primary Label

● Input: entities / labels with different span alternatives● Task: for a label with multiple span alternatives, choose the primary ones● Template:

● Result: ranked span alternatives for label

13


Crowd - Determine Label Relevance

● Input: list of labels● Task: for a given label, determine whether the label is relevant for the

synopsis● Template:

● Result: ranked entities based on their relevance and completeness

14


Crowd - Determine Label Type

● Input: labels with no type or type - disagreement among machines● Task: choose the type of each highlighted label ● Template:

● Result: label type

15


Crowd - Extracting Events (1)

● Input: one video description● Task: highlight all the events from the video description● Template:

● Result: list of possible events

16


Conclusions

● difficult to find one NER tool that performs well

● combining the output of several NER tools results in disagreement

● by applying CrowdTruth to machine disagreement we were able to identify possible problems of the machine output

● by applying CrowdTruth on the information gathered from the crowd we were able to identify:○ the entity span○ the entity type○ the entity relevance score○ possible events in the video description

17

harnessing the power of machines & crowds for event extraction

Data & Analytics