Crowd Truth Machine-Human Computation Framework for Harnessing Disagreement in Semantic Interpretation

Goal: gather ground truth data to train, test, evaluate cognitive computing systems

Humanannotationtasks:

use case #1: text annotation

concept: drug

concept: disease

relation:

treats

use case #2: image annotation use case #3: video annotation

Batavia

Army

Military

Indonesia

Triangle ofdisagreement

Oana Inel, Khalid Khamkham, Tatiana Cristea, Arne Rutjes, Jelle van der Ploeg, Lora Aroyo,Robert-Jan Sips, Anca Dumitrache and Lukasz Romaszko

crowdsourcing task

sentence vector

unit vectors for the same sentence

unit vector

sentence-annotation score

● sentence-annotation score:measures how clearly the annotation is expressed in the sentence

● sentence clarity:measures the maximum sentence-annotation score for the sentence

VS. The CrowdTruth approach

ask a large crowd

● allows for different interpretations● minimal instructions● large crowds of annotators● harnessing disagreement● continuously updated with new data

Traditional Ground Truth approach

ask few experts

● assumes one correct interpretation● guidelines limit interpretations● examples evaluated by single expert● eliminating disagreement● ground truth reused over time

Approach:

● worker-sentence score: measures quality of worker for one sentence

● worker-worker disagreement:● measures pairwise agreement between workers● average worker agreement:

measures overall worker quality

worker-sentence score

Disagreementmetrics:

(use case #1:text annotation)