crowdsourcing for watson

Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo

Crowdsourcing for Watson Lora Aroyo

Monday, December 10, 12

Observationlanguage is ambiguous & vague

people interpret it from different points of view human disagreement may signal

ambiguity or vagueness of language

Flickr: elkabong Monday, December 10, 12

our goal is ...

not to enforce agreementto capture different view points

adapt to different annotation tasks

Flickr: elkabongMonday, December 10, 12

so we need to ...

understand the range of disagreements by creating a space of possibilities with frequencies & similarities

teach machines to reason in the disagreement space score their output based on where it falls in this space


Harnessing the Crowd

Flickr: AmyJanelle

not just to save money


http://www.flickr.com/photos/30868936@N08/

http://www.flickr.com/photos/30868936@N08/

Worker #1 Answer

?

?

?

MICRO-‐TASK

Worker #2 Answer

Worker #3 Answer

Worker #N Answer

?


Lora Aroyo

The Task Answers


Lora Aroyo

OLANZAPINE is an atypical antipsychotic, approved by the U.S.Food and Drug Administration (FDA) for the treatment of SCHIZOPHRENIA and bipolar disorder.

?OLANZAPINEIs SCHIZOPHRENIA related to

treated_by treats may_treat

?RESPIRATORY ALKALOSIS

Is HYPERVENTILATION related to

cause_of cause symptom_ofdiagnosed_by

RESPIRATORY ALKALOSIS is a medical condition in which increased respiration (HYPERVENTILATION) elevates the blood pH (a condition generally called alkalosis).


Lora Aroyo

OLANZAPINE is an atypical antipsychotic, approved by the U.S.Food and Drug Administration (FDA) for the treatment of SCHIZOPHRENIA and bipolar disorder.

?OLANZAPINEIs SCHIZOPHRENIA related to

treated_by treats may_treat

?RESPIRATORY ALKALOSIS

Is HYPERVENTILATION related to

cause_of cause symptom_ofmay_causediagnosed_by

RESPIRATORY ALKALOSIS is a medical condition in which increased respiration (HYPERVENTILATION) elevates the blood pH (a condition generally called alkalosis).


Lora Aroyo

He was the first physician to identify the relationship between HEMOPHILIA and HEMOPHILIC ARTHROPATHY.

?HEMOPHILIC ARTHROPATHY

Is HEMOPHILIA related to

treated_by may_causeother

cause_of cause

has_manifestation

symptom_of

It just says there is a relation between the two but gives no specifics about what the relation is

There is a relationship between the two disorders but the sentence does not indicate what that relationship is.

identify the relationship between


Lora Aroyo

Sentence-Relation Distribution


Lora Aroyo

Expert Annotators


Lora Aroyo

Crowd vs. Experts


Lora Aroyo

Crowd vs. Experts

x


The Dark Side of Crowdsourcing Disagreement

• disagreement is beautiful, except when it results from spamming• crowdsourcing has to account for people that want to get paid for

not doing any work• spammers generate disagreement for the wrong reasons• most spam detection requires gold standard


PositionArtificially restricting humans does not help machines to learn.

Machines will learn from diversity


JOIN our team• Dr. Watson

Medical Synonyms Game

• Medical DetectiveGamification of Medical Diagnosis

• To CATCH a ThiefSPAM detection in crowdsourcing with no Gold Standard

[email protected]

choose Your

Thesis project

call now:


http://watsonmedicaldetective.wordpress.com/




mailto:[email protected]




crowdsourcing for watson

Technology